Simpletransformers: Problems when classifying after finetuning BERT (Multi-Label)

[ErikTromp]

I am following the write-up to a muti-label classification as done here https://towardsdatascience.com/multi-label-classification-using-bert-roberta-xlnet-xlm-and-distilbert-with-simple-transformers-b3e0cda12ce5

I am having some difficulties. I loaded a Dutch base BERT model (from here https://github.com/wietsedv/bertje) and then I train a multi-label model with 50 labels:

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv("all_data_withid.csv", encoding="utf8", delimiter=";")
df['labels'] = list(zip(df.label1.tolist(), df.label2.tolist(), ...)) #truncated for brevity
train_df, eval_df = train_test_split(df, test_size=0.3, random_state=123456)

model = MultiLabelClassificationModel('bert', 'bert-base-dutch-cased/bertje-base', num_labels=50, args={'train_batch_size':2, 'gradient_accumulation_steps':16, 'learning_rate': 3e-5, 'num_train_epochs': 1, 'max_seq_length': 512, 'fp16': False})

result, model_outputs, wrong_predictions = model.eval_model(validation_df)

Now the end result is that I get an LRAP score of roughly 0.71. However, now I am a bit puzzled on how to use this model to classify a single new instance. I closed Python, opened it again and loaded my trained model from disk:

model = MultiLabelClassificationModel('bert', 'outputs', num_labels=50, args={'train_batch_size':2, 'gradient_accumulation_steps':16, 'learning_rate': 3e-5, 'num_train_epochs': 1, 'max_seq_length': 512, 'fp16': False}).

I then tried model.predict(["dit is een test"]) and model.predict(["en nog een compleet andere test"])
and as it turns out the resulting outputs and predictions (always all 0s for every class) for these 2 distinct sentences are exactly the same on all values. I also tried to evaluate (result, model_outputs, wrong_predictions = model.eval_model(validation_df)) 3 times on different splits of my dataset but in all scenarios the resulting LRAP is the same ~0.71.

What am I doing wrong here?

[ThilinaRajapakse]

You aren't doing anything wrong. I found a bug that was introduced in a recent update which was causing the predict function to use cached features. This should be resolved now. Please let me know if the issue persists.

[ErikTromp]

Seems to change the behavior indeed. Note for others: I did have to remove the cache dir.

[ThilinaRajapakse]

Is it still buggy if you don't remove the cache dir?

[ErikTromp]

Yes, what I tried first (after updating) was:

from simpletransformers.classification import MultiLabelClassificationModel
model = MultiLabelClassificationModel('bert', 'bert-base-dutch-cased/bertje-base', num_labels=50, args={'train_batch_size':2, 'gradient_accumulation_steps':16, 'learning_rate': 3e-5, 'num_train_epochs': 1, 'max_seq_length': 512, 'fp16': False, 'evaluate_during_training': True})

Then I threw some examples at it and they all ended up producing the same output. Then I removed the cache dir and executed the same code after which I got distinct outputs.

I am now going to finetune my classifier again based on this model. I will report back if it turns out to be static or wrong again.

[RORO-ML]

Have the same problem, removing the cache dir doesn't fix it.

model = MultiLabelClassificationModel(
    'bert',
    'bert-base-cased',
    num_labels=lable_num,
    args={"reprocess_input_data': True,
          'overwrite_output_dir': True, 
          'num_train_epochs': 6, 
          'use_cached_eval_features' : False, 
          'save_model_every_epoch':False,
          'train_batch_size':  13,
          'max_seq_length': 256,
          'no_cache': True,
          },
)

Can't figure out why, am I doing something wrong?

[ErikTromp]

Just stating the obvious: you did update to latest version, right?

[RORO-ML]

Rechecked it, its the newest (0.20.3)

[RORO-ML]

Thought it could be because I have over 1k labels, but even then it should work normally (at least change the scores slightly)

[ErikTromp]

Hmm well that is not what I am experiencing. Since you are using BERT as a base model - what happens if you run the 1k classifier on data without finetuning? Outcomes should make no sense but at least differ given different inputs (wasn't true for me before I updated and removed the cache, although I am loading a custom BERT model from disk in Dutch) and it should be fairly easy to check.

[RORO-ML]

Do you mean to test it on a not pretrained?

I will check it on another dataset and give you feedback - and I will check other models as base as well. The plan was to try out if something does change and then overwrite the classifiers forward to improve the results (or at least get them to make some sort of sense). I was hoping for behavior like after a 1d CNN.

[ThilinaRajapakse]

The outputs should be different for different outputs using any model although there is some instability in certain cases. Training a model for too long (particularly the large models) seems to break the model and give the same output (or close enough) for all inputs. The easiest way to avoid this is to use early stopping.

[RORO-ML]

Thank you both 👍

I will retry it and post the results. I just trained the base model over 6 epochs, so it shouldn't overfit that much.

[ThilinaRajapakse]

6 epochs is actually a substantial amount of training in the context of fine-tuning.

You can also see #199 for a discussion on the instability issue.

[Lysimachos]

I have the same issue #231.
I updated the simpletransformers library and deleted the cache_dir but the problem remains:
The predictions are 0 for all labels. I used the following:

self._model = MultiLabelClassificationModel('bert', 'bert-base-uncased', num_labels=90, args={'train_batch_size': 32, "eval_batch_size": 32, 'gradient_accumulation_steps': 4, 'num_train_epochs': 3, "max_seq_length":256, "use_multiprocessing": False, "reprocess_input_data": True, "use_cached_eval_features": False, 'overwrite_output_dir': True,})

[ErikTromp]

That, I have too, all predictions are always 0. Outputs do change tho

[ThilinaRajapakse]

Can you guys post your training_progress_scores.py training_progress_scores.csv (and preferably a graph of loss)? If the outputs are different, I don't think it's a caching issue anymore.

[ErikTromp]

I am not sure what you mean with that Python file. Training a single epoch for me takes 5-10 hours depending on batch size and what not so running multiple epochs with evals in between is pretty time consuming.

It's pretty easy to reproduce though, I just downloaded a Dutch BERT model (https://github.com/wietsedv/bertje) and train it on some data:

from simpletransformers.classification import MultiLabelClassificationModel
model = MultiLabelClassificationModel('bert', 'bert-base-dutch-cased/bertje-base', num_labels=47, args={'n_gpu': 1, 'train_batch_size':8, 'gradient_accumulation_steps':16, 'learning_rate': 1e-4, 'num_train_epochs': 1, 'max_seq_length': 256, 'fp16': False, 'reprocess_input_data': True, 'use_cached_eval_features': False, 'evaluate_during_training': True, 'evaluate_during_training_verbose': True, 'output_dir': 'outputs/'})

Any small dataset would do I guess but mine contains 600k examples. All predictions always end up being 0 for every example given, even those from the training set.

PS. I also noticed some other oddities, like fp16 for me is 25% slower than not using it. Using GPU is also ~5% slower than using my GPU (a 980m). Not sure if that hints to anything wrong in my setup explaining this.

[ThilinaRajapakse]

My bad, it's training_progress_scores.csv.

You could use evaluate_during_training and give a very small eval_df so that it won't significantly impact the overall time taken. I understand that it can still be frustrating. If you can give me a small Dutch dataset (I don't speak Dutch), I can see what I can do on my end.

I'm not a 100% sure on this, but using fp16 could possibly slow down training on non-RTX cards (or cards without tensor cores optimized for half-precision calculations). In that case, the memory consumption should be lower, but training will be slower.

Using GPU is also ~5% slower than using my GPU (a 980m).

Do you mean that the model trains faster on the CPU than on the GPU? That is _definitely_ off.

[ErikTromp]

Sorry, I am also stumbling over my own feet. The GPU is (a lot) faster, I just hadn't used the use_cuda flag in the right place as per my other issue. For fp16, I also read it can slow things down if batch_size is pretty low and you have a relatively old GPU (both true for me).

As for the actual issue with predictions being 0 all the time, I may have found a small culprit on my own end that I am examining now, will report back in a few hours. In the meantime, this is a very small sample of what I am using, with the code I am using. The code may look a bit odd because I am preprocessing the data a little bit and what I dumped in the CSV is already preprocessed slightly, though it should still work. https://gist.github.com/ErikTromp/6c904e64386114566c3e0c1ed2abe424

[ThilinaRajapakse]

I am also rerunning my code for the multi-label article I wrote on Github to see whether something broke in an update between then and now. Let's check back in when we have more info.

Thanks for your support in bug hunting!

[ErikTromp]

So, I ran one epoch, see ouput below.

training_progress_scores.csv

global_step,LRAP,train_loss,eval_loss
911,0.4679538123143837,0.058400850743055344,0.08789259699165822

Prediction for two very distinct cases that should produce radically different classes (though perhaps not after 1 epoch):

from simpletransformers.classification import MultiLabelClassificationModel
model = MultiLabelClassificationModel('bert', 'outputs', num_labels=47, args={'train_batch_size':8, 'gradient_accumulation_steps':16, 'learning_rate': 1e-4, 'num_train_epochs': 5, 'max_seq_length': 256, 'fp16': False, 'reprocess_input_data': True, 'use_cached_eval_features': False, 'evaluate_during_training': True, 'evaluate_during_training_verbose': True, 'output_dir': 'outputs2/'})

>>> model.predict(['dit is een goede test'])
Converting to features started. Cache is not used.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.35s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.22it/s]
([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], array([[0.00490209, 0.00475683, 0.0064099 , 0.0066854 , 0.00475818,
        0.00583799, 0.00740596, 0.00727157, 0.00572323, 0.00910022,
        0.00645337, 0.00594897, 0.00752376, 0.00840348, 0.00930631,
        0.00861833, 0.00988511, 0.00804399, 0.00796497, 0.00741255,
        0.01309202, 0.00617968, 0.00939393, 0.00891096, 0.00560355,
        0.01215368, 0.01360207, 0.01089549, 0.01267053, 0.01134402,
        0.01008886, 0.01287259, 0.00939571, 0.01785794, 0.0170364 ,
        0.01176586, 0.01493526, 0.02007966, 0.03211075, 0.02868371,
        0.00919858, 0.03075826, 0.03692437, 0.04875872, 0.06632923,
        0.02507584, 0.10394432]], dtype=float32))
>>> model.predict(['dit is een slechte test'])
Converting to features started. Cache is not used.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.34s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.44it/s]
([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], array([[0.00570513, 0.00461973, 0.00799095, 0.00717648, 0.00426016,
        0.00576732, 0.00803969, 0.00743552, 0.0055523 , 0.01165319,
        0.0064672 , 0.00539476, 0.00880006, 0.0116614 , 0.0128232 ,
        0.01045122, 0.01252311, 0.00845576, 0.00731844, 0.00733786,
        0.01712303, 0.00630884, 0.00888824, 0.00880629, 0.00481787,
        0.01580676, 0.01269947, 0.00995326, 0.0154126 , 0.01244096,
        0.009538  , 0.01376155, 0.00773394, 0.02110994, 0.02301501,
        0.01082647, 0.01241072, 0.02995057, 0.0545058 , 0.02899898,
        0.00726688, 0.0240171 , 0.03071511, 0.07382316, 0.07297762,
        0.02195006, 0.16699038]], dtype=float32))

[djSharma7]

Hi Erik,
Have you tried passing some string in predict method, which is also present in the training set with which you have trained your model!
If not can you pick out some strings out from the training set and try them with the predict function and then share the results!

[ErikTromp]

Yes I have, training examples also result in all 0s for predict. I also noticed the values on the outputs hardly change (though they do change) for any example I predict on.

[ErikTromp]

I'll get back to basics and just use the NL BERT model with a handcrafted, small dataset and see how that works out. I will do this later today when I have time - will report back in.

[Lysimachos]

Here is my finding over this issue. I use a standard data (Reuters from nltk) for multi-label classification.

self._train_df = pd.DataFrame(OrderedDict({'text': self._data_train[DOCUMENTS_IDENTIFIER],
                                                    'labels': self._data_train[CLASSES_IDENTIFIER]}))
self._eval_df = pd.DataFrame(OrderedDict({'text': self._data_val[DOCUMENTS_IDENTIFIER],
                                                 'labels': self._data_val[CLASSES_IDENTIFIER]}))

BertForSequenceClassification.from_pretrained('bert-base-uncased', force_download=True)
self.model = MultiLabelClassificationModel('bert', 'bert-base-uncased', num_labels=90, args={'train_batch_size': 32, "eval_batch_size": 32,
                                                                                                      'gradient_accumulation_steps': 4,
                                                                                                      "evaluate_during_training":True,
                                                                                                      'num_train_epochs': 2, "max_seq_length":256,
                                                                                                      "use_multiprocessing": False,
                                                                                                      "reprocess_input_data": True,
                                                                                                      "use_cached_eval_features": False,
                                                                                                      'overwrite_output_dir': True,
                                                                                                      'evaluate_during_training_verbose': True})

        # Train the model
        self.model.train_model(self._train_df,eval_df=self._eval_df)

        # Evaluate the model
        result, model_outputs, wrong_predictions = self.model.eval_model(self._eval_df)
        print(result)

.....
Running loss: 0.167303
Running loss: 0.170446
Running loss: 0.164527
Running loss: 0.164525
Running loss: 0.167300
Running loss: 0.175209
Running loss: 0.171032
Running loss: 0.170071
Running loss: 0.175808
Running loss: 0.170429
Running loss: 0.168178
Running loss: 0.177569
Running loss: 0.168423
Running loss: 0.174538
Current iteration: 100%|██████████████████████| 240/240 [01:01<00:00, 4.13it/s]Converting to features started. Cache is not used.
{'LRAP': 0.609899021999634, 'eval_loss': 0.16782281175255775}
Epoch: 100%|██████████████████████████████████████| 2/2 [02:13<00:00, 67.58s/it]
Training of bert model complete. Saved to outputs/.
100%|████████████████████████████████████████| 100/100 [00:00<00:00, 161.75it/s]
100%|█████████████████████████████████████████████| 4/4 [00:00<00:00, 10.71it/s]
{'LRAP': 0.609899021999634, 'eval_loss': 0.16782281175255775}

The training_progress_scores.csv:

global_step | LRAP | train_loss | eval_loss
-- | -- | -- | --
60 | 0.462933835124969 | 0.241688281297684 | 0.23544592782855
120 | 0.609899021999634 | 0.174538105726242 | 0.167822811752558

So far everything looks as it should. The loss function started from around 0.69 in first iteration in epoch 1 and went to around 0.17 in last iteration of epoch 2. The loss of the eval_df (100 examples) follows the loss function of the training sample and outputs the exact same eval_loss either calculated from model.train_model or model.eval_model. Now I used as test data the eval data:

        outputs, raw_outputs = self.model.predict(self._data_val[DOCUMENTS_IDENTIFIER])
        print(raw_outputs)
        print("min_value: ", raw_outputs.min(), " max_value: ", raw_outputs.max())
        outputs, raw_outputs = self.model.predict([self._data_val[DOCUMENTS_IDENTIFIER][0]])
        print(raw_outputs)
        print("min_value: ", raw_outputs.min(), " max_value: ", raw_outputs.max())
        outputs, raw_outputs = self.model.predict([self._data_val[DOCUMENTS_IDENTIFIER][1]])
        print(raw_outputs)
        print("min_value: ", raw_outputs.min(), " max_value: ", raw_outputs.max())
        outputs, raw_outputs = self.model.predict([self._data_val[DOCUMENTS_IDENTIFIER][56]])
        print(raw_outputs)
        print("min_value: ", raw_outputs.min(), " max_value: ", raw_outputs.max())

Converting to features started. Cache is not used.
100%|████████████████████████████████████████| 100/100 [00:00<00:00, 564.63it/s]
100%|█████████████████████████████████████████████| 4/4 [00:00<00:00, 9.67it/s]
[[0.33642578 0.23144531 0.1496582 ... 0.09753418 0.16137695 0.10266113]
[0.33618164 0.23754883 0.14941406 ... 0.09667969 0.16259766 0.10339355]
[0.33764648 0.23474121 0.14929199 ... 0.09686279 0.16235352 0.10302734]
...
[0.33935547 0.23376465 0.1484375 ... 0.09667969 0.16210938 0.10339355]
[0.33764648 0.23461914 0.1496582 ... 0.09667969 0.16296387 0.10339355]
[0.33862305 0.23828125 0.14892578 ... 0.09667969 0.16259766 0.10412598]]
min_value: 0.07989502 max_value: 0.35009766
Converting to features started. Cache is not used.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 128.48it/s]
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 51.46it/s]
[[0.33642578 0.23144531 0.1496582 0.1529541 0.15686035 0.14489746
0.13476562 0.11737061 0.12683105 0.14074707 0.17529297 0.16369629
0.16174316 0.1116333 0.13647461 0.14953613 0.19555664 0.18713379
0.11920166 0.14697266 0.12597656 0.11340332 0.17260742 0.13916016
0.14562988 0.13952637 0.11083984 0.09387207 0.1550293 0.10070801
0.12036133 0.13464355 0.10559082 0.12976074 0.15161133 0.20166016
0.14855957 0.140625 0.18493652 0.09844971 0.140625 0.19641113
0.203125 0.10015869 0.12792969 0.13256836 0.10467529 0.13867188
0.14587402 0.21606445 0.10705566 0.15405273 0.14916992 0.140625
0.1538086 0.10302734 0.12512207 0.11993408 0.10015869 0.20544434
0.0927124 0.10968018 0.1295166 0.0803833 0.09533691 0.15710449
0.18041992 0.1451416 0.09533691 0.12127686 0.11993408 0.08648682
0.10949707 0.1529541 0.14086914 0.11920166 0.16467285 0.14587402
0.14453125 0.0881958 0.12927246 0.11474609 0.13146973 0.15063477
0.1706543 0.10949707 0.12792969 0.09753418 0.16137695 0.10266113]]
min_value: 0.0803833 max_value: 0.33642578
Converting to features started. Cache is not used.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 257.05it/s]
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 51.32it/s]
[[0.33618164 0.23754883 0.14941406 0.1508789 0.1574707 0.1427002
0.13476562 0.11657715 0.12683105 0.13989258 0.17260742 0.1607666
0.16186523 0.11065674 0.13452148 0.1484375 0.19763184 0.18908691
0.11755371 0.14855957 0.12445068 0.11199951 0.16967773 0.13903809
0.14550781 0.140625 0.10913086 0.09448242 0.15441895 0.1015625
0.11920166 0.13513184 0.10430908 0.12890625 0.15136719 0.20324707
0.14697266 0.14367676 0.18383789 0.09857178 0.14343262 0.19787598
0.20373535 0.10144043 0.12780762 0.13244629 0.10449219 0.13842773
0.14660645 0.21838379 0.10839844 0.15490723 0.1484375 0.1385498
0.15368652 0.10144043 0.12561035 0.11859131 0.09912109 0.20751953
0.09222412 0.11047363 0.13195801 0.08105469 0.0960083 0.15917969
0.17993164 0.14489746 0.09283447 0.12011719 0.1184082 0.08447266
0.10968018 0.15112305 0.14001465 0.11981201 0.16479492 0.14587402
0.14257812 0.08898926 0.12927246 0.11657715 0.1315918 0.15063477
0.17297363 0.10778809 0.12780762 0.09667969 0.16259766 0.10339355]]
min_value: 0.08105469 max_value: 0.33618164
Converting to features started. Cache is not used.
100%|███████████████████████████████████████████| 1/1 [00:00<00:00, 1286.60it/s]
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 51.49it/s]
[[0.34936523 0.2319336 0.14941406 0.15112305 0.15710449 0.14404297
0.13537598 0.11737061 0.12432861 0.13964844 0.17468262 0.16235352
0.16296387 0.11047363 0.13439941 0.14855957 0.1973877 0.18786621
0.11676025 0.1496582 0.12451172 0.1116333 0.16918945 0.14025879
0.1459961 0.13928223 0.11126709 0.09417725 0.15539551 0.09875488
0.12011719 0.13586426 0.10540771 0.13000488 0.1508789 0.20153809
0.15039062 0.14172363 0.1850586 0.097229 0.14099121 0.19873047
0.20178223 0.10070801 0.12768555 0.1373291 0.10321045 0.14123535
0.14782715 0.21582031 0.10742188 0.15515137 0.15014648 0.13830566
0.15270996 0.10211182 0.12585449 0.1159668 0.0993042 0.20788574
0.09332275 0.11279297 0.13061523 0.07989502 0.09533691 0.15808105
0.18054199 0.14880371 0.09234619 0.12011719 0.11981201 0.08557129
0.10797119 0.15234375 0.13891602 0.11859131 0.16540527 0.14953613
0.14221191 0.08911133 0.12927246 0.11633301 0.12780762 0.15124512
0.17382812 0.10760498 0.12854004 0.0970459 0.16174316 0.10394287]]
min_value: 0.07989502 max_value: 0.34936523

The outputs are not that different - they are really really close to each other (the eval data where taken after a lot of shuffle). I also tried some instances from test data. All the predictions were like the above.
Btw if I do all the above except from the training step, the untrained model (as given from hugging face) makes predictions more than 0.5 and quite different from each other.

[ThilinaRajapakse]

I tested with the Toxic Comments dataset (guide here). I trained a Multilabel Roberta model with the arguments given below. The reason for the tiny batch size is because my primary GPU was working on something else at that time, but I don't think this should change anything.

train_args = {
    'reprocess_input_data': True,
    'train_batch_size':2, 
    'gradient_accumulation_steps':8, 
    'learning_rate': 3e-5, 
    'num_train_epochs': 1, 
    'max_seq_length': 512, 
    'save_steps': 500000, 
    'overwrite_output_dir': True,
    'threshold': 0.5,
    'fp16': False
}

Then I tested with the following script.

from simpletransformers.classification import MultiLabelClassificationModel
import pandas as pd
import numpy as np
from pprint import pprint


model = MultiLabelClassificationModel('roberta', 'outputs', num_labels=6, args={'max_seq_length': 512, 'eval_batch_size': 16})

test = {
    "text": ["You are the worst person ever.", "You are the best person ever."],
    "labels": [[1, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 0]]
}

test_df = pd.DataFrame.from_dict(test)
print("--- Test DataFrame ---")
print()
print(test_df.head())
print("------\n")

print("--- Legend ---")
print("The labels are toxic, severe_toxic, obscene, threat, insult, identity_hate respectively.")
print("------\n")

results, out, _ = model.eval_model(test_df)
print("--- Eval Results ---")
pprint(results)
pprint(out)
print("------\n")

print("--- Predictions ---")
print()
print("'You are the worst person ever.'")
pprint(model.predict(['You are the worst person ever.']))
print()
print("'You are the best person ever.'")
pprint(model.predict(['You are the best person ever.']))

This is the output.

--- Test DataFrame ---

                             text              labels
0  You are the worst person ever.  [1, 0, 0, 0, 1, 0]
1   You are the best person ever.  [0, 0, 0, 0, 0, 0]
------

--- Legend ---
The labels are toxic, severe_toxic, obscene, threat, insult, identity_hate respectively.
------

--- Eval Results ---
{'LRAP': 1.0, 'eval_loss': 0.05164750665426254}
array([[9.30425465e-01, 6.08645193e-03, 1.20048076e-01, 4.90952050e-03,
        6.78570986e-01, 1.62411109e-02],
       [3.38225882e-03, 1.22283454e-04, 3.02517350e-04, 1.43892641e-04,
        4.53031302e-04, 1.91567160e-04]], dtype=float32)
------

--- Predictions ---

'You are the worst person ever.'
Converting to features started. Cache is not used.
([[1, 0, 0, 0, 1, 0]],
 array([[0.93042547, 0.00608645, 0.12004808, 0.00490952, 0.678571  ,
        0.01624111]], dtype=float32))

'You are the best person ever.'
Converting to features started. Cache is not used.
([[0, 0, 0, 0, 0, 0]],
 array([[0.00338226, 0.00012228, 0.00030252, 0.00014389, 0.00045303,
        0.00019157]], dtype=float32))

It's not giving the same (or similar) outputs from what I can see. :thinking:

[ErikTromp]

Okay so I tried quite a lot of combinations to see when I do and do not get sensible results (or well: at least one non-zero vs only zeros). What I tried:

• Load the custom Dutch BERT from disk
• Load the custom Dutch BERT model from the Transformers repo (they're actually uploaded there)
• Try a Roberta model
• Use all labels, in my case 47
• Use less labels (I went with 5)
• Tried a larger learning rate

In all of these scenarios, I am getting all 0s as predictions. Even when I feed it examples that are in the training set.... All outputs are also all pretty close to one another.

I am now trying to wrap my head around whether this makes sense or not (it shouldn't, right?). I also ran 10 epochs on a smaller sample of my dataset for time's sake and checked the training and evaluation loss together with LRAP after every epoch: it seems there is hardly any improvement after epoch 4 which makes me think there is no such thing as vanishing gradients or whatnot going on.

Concluding: I still can't figure out why I am seeing this behavior, it makes me think it's due to Huggingface's Transformers instead of this library. If I find the time, I might use their run_glue.py script on my dataset and see how that performs.

Any other thoughts? Am I just rambling here?

[ThilinaRajapakse]

I think this information sheds some light on the situation.

The common theme I am seeing here is that there are a large number of labels in most cases. When you have a large number of labels, the model does tend to predict 0's (or close to 0) for everything. The reason is, in most datasets, you tend to get a lot more 0's for each label individually compared to the number of 1's you have for the same label.

The reason that it works for me with the Toxic Comments dataset is probably that are only 6 labels in that dataset (and fairly well balanced if I remember correctly).

Also, the LRAP score can be a little confusing in this context as well. It does not care about the _distance from prediction to ground truth_ as with most rankings, and considers rather the relative ranking of the labels. For example,

>>> lrap([[1., 0.]], [[0.99, 0.01]])
1.0
>>> lrap([[1., 0.]], [[0.1, 0.01]])
1.0
>>> lrap([[1., 0.]], [[0.02, 0.01]])
1.0

As long as the relative ranking of the labels is correct, LRAP is perfectly happy to give a perfect score.

Since you guys are getting good LRAP scores, I believe the model is actually learning the task but is getting tripped up by the threshold.

You could try lowering the threshold argument (default is 0.5). Maybe change it to the average of the model predictions? I know this is quite hacky but if it improves the prediction, I think it lends quite a bit of credibility to my hunch.

_Edit_
_P.S. Threshold is the value used as the separator between positive and negative samples._

[ErikTromp]

Yes I was thinking along those same lines as you are. However, I can't find evidence of that being the case, in fact, the contrary.

As you can tell by now, I am trying to predict emojis that occur in tweets by using transformers (much like https://github.com/bfelbo/DeepMoji except they don't use transformers).

To test whether fewer classes made any difference, I took my full dataset but instead of adding all 47 emojis to the labels list, I only took the 5 most frequently occurring ones. The result was still predictions with all 0s, regardless of input (even from the training set).

What I had already done before to test this, was to look at the outputs instead of the predictions. I figured that if the outputs at least scaled for the emojis I was trying to predict, I could make something out of it. Unfortunately however, if I fed the model a tweet it had seen during training with a 'joy' emoji (highly positive), it would produce some outputs, then if I fed it another tweet with a 'pout' emoji (highly negative) it would produce seemingly random outputs (even higher score for 'joy' and lower for 'pout').

It seems that the model isn't capable of learning anything here. Only thing (apart from a fundamental flaw I am missing or doing wrong (which is still likely)) I can think of is that I have a dataset that simply cannot be modeled this way. Problem I have there is that DeepMoji has shown it is perfectly possible (albeit in a different language) and the paper from the Dutch roberta model (https://arxiv.org/pdf/2001.06286.pdf) show that it works very well on downstream sentiment analysis.

Note also - my initial LRAP was 0.71 but I realized I had far too many examples without emojis. I downsampled that to about 10% (equal in size to texts that have the most frequently occurring emoji: joy) and now my LRAP doesn't go beyond 0.44.

[ErikTromp]

Maybe we are running into https://github.com/huggingface/transformers/issues/2263 or https://github.com/huggingface/transformers/issues/1465 ?

[ErikTromp]

Ok so here I am again with a more detailed report.

I have been trying out several things to get to the bottom of this but long story short: I haven't...

What I tried was using less classes, 3 for example. This seems to produce no more meaningful results than any higher number of classes. I also tried using auto_weights: True but that didn't help much either.

The only scenario in which I was able to produce 1s in the prediction was if I filtered my dataset such that at least one of the labels contains a 1, ie. filtering out all records which contain no emojis in my case. This works when I have little data (ie. very dense data) but it does not work if I use my 47 labels where I keep all records with at least one 1... I then still get predictions with only 0s.

Note that also the outputs do not correspond well with the input data. If I run model.predict on records from my training set, neither the predictions nor the outputs come even close to what the original labels were....

It seems that this way of doing multilabel classification is extremely sensitive to class imbalance (which I have a great deal). This seems to make no sense to me at all either because the toxic dataset contains all 0s for their labels in almost 90% of all records in their training set....

[ThilinaRajapakse]

I am at a loss at this point. The only explanation I can think of is that because of the imbalance between 0s and 1s, the model doesn't even bother (or is incapable of) learning even which labels should be closer to 1s.

This might not give much additional insight, but what if you simplified the problem a lot more by changing it to a simple multiclass sentiment analysis problem? You could assign a sentiment to the emoji like with Sentiment 140 dataset. At the least, it would confirm that the issue is not with the Dutch Bert model or the dataset.

Also, try with much lower learning rate (1e-6)

[ErikTromp]

So am I. Your idea is worth a shot, I will try and do that and then compare it to a fasttext/doc2vec for comparison.

Will report back in.

[ErikTromp]

Finally reporting back in. What I did was turn all the emojis into a number of classes and made it a multiclass rather than multilabel problem. I now have 8 classes for emojis (one for each of Plutchik's model: https://en.wikipedia.org/wiki/Robert_Plutchik#Plutchik%27s_wheel_of_emotions) and a 9th class being no_emotion.

I trained a normal ClassificationModel using SimpleTransformers and I am getting nowhere better than the majority class baseline (roughly 47% in my case). I also tried other Transformer-based libraries but I had massive difficulties getting them to work in the first place (tried Spacy-Transformers, ktrain and this vanilla notebook https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/6%20-%20Transformers%20for%20Sentiment%20Analysis.ipynb). Only got the notebook to run which produced pretty terrible results too.

All of this was done using the Dutch BERT model that in their paper has proven very effective to predict emotions. On the other hand, DeepMoji (https://deepmoji.mit.edu/) has shown results that predicting emotions/sentiment from emojis is a valid way. I tried to tie the two together by using the Dutch BERT model to predict emotions by using emojis.

The end result is pretty disappointing, either because my dataset (NL tweets) is not well suited for this (DeepMoji begs to differ) or because the Dutch BERT model is not well suited for this (their work begs to differ - https://arxiv.org/abs/1912.09582). Concluding: I do not understand this but it seems unrelated to SimpleTransformers.

Finally I ran a supervised classifier based on FastText's pretrained Dutch word embeddings and their scripts for training a classifier. I used the same dataset as what I used for all transformer-based experiments. It's multiclass now instead of multilabel so I am getting:

N       83577
P@1     0.61
R@1     0.61

These results are far better than what I've seen with transformers though not as promising as DeepMoji claims at all. It does hint slightly to me thinking that the Dutch BERT model is off somehow.... Mind you though that my data is very heavily imbalanced wrt classes which might be worth further investigating.

[ThilinaRajapakse]

Just a hunch, but could it be a weird behaviour produced by the _combination_ of Dutch BERT models pretrained embeddings _and_ emojis (and the way emojis get tokenized/represented)? It seems quite far-fetched but you never know.

Did you try using the Dutch BERT model on another (standard) dataset with Simple Transformers?

[ErikTromp]

Well I take all the emojis out of the training data as they are the actual labels so them being OoV shouldn't matter much. I have tried NL BERT on other sets but not on standard ones yet as there aren't many NL datasets. I can try give the paper that produced NL BERT and their datasets a go. I will report back in.

[ErikTromp]

Okay so, some more tests I tried:

• Balance the data to remove class imbalance
• Run a classifier using FastText vs BERT-based

What I observed was that when I balanced the classes out (9 classes, so ~11% for each), the result using BERT was about 30%. Using FastText with imbalanced classes as well as using balanced classes yielded subpar results, 28.8% for the balanced case for example.

I also ran a classifier using NL BERT on the Dutch 110k book review dataset, used in this paper https://arxiv.org/abs/2001.06286. They show both a Dutch roBERTa model and the BERT model I am using. The BERT model scores 93% accuracy on this dataset.

I tried to gather as much of the parameters as used in the paper (max_seq_length set to 128, ran for 1 epoch, batch_size 4, learning rate 1e-5). The result I obtained is an accuracy of 79.3%, far, far lower than what they obtained in their paper. Surely my paramters might be off by a bit, but not this much....

Concluding:

• It seems that, despite papers showing otherwise, using pre-trained embeddings on Twitter data to predict emojis or emotions is not really possible in Dutch.
• I am unable to reproduce results shown in papers by a large margin.

I do wonder this though: DeepMoji seems to also learn the word embeddings based on their classification task. Does SimpleTransformers freeze the n-1th layer, thus only learning the weights for the classification task? Or does it propagate back to the actual embeddings too?

All in all, I don't think my quest for the truth has much to do with SimpleTransformers but more so my application thereof. Also - I am so far very much underwhelmed by transformers in general as living up to the papers' results seems far from reality (due to whatever case, even if just my wrong application thereof).

[Lysimachos]

So I have done some progress in finding out what's going on.

I have noticed two issues:
1) The final layer - the classification layer does not get the most appropriate inputs. Take bert classification model (class: BertForMultiLabelSequenceClassification in file: simpletransformers/custom_models/models.py):

outputs = self.bert(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
        )

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

outpus[1] is a tensor [batch_size, hidden_size] which only includes the attention weights to each of the embeddings dimensions NOT the embeddings. A better way to would be either to get the [CLS] embedding of each text, or all word embeddings. So I propose to change the code in one of the adove ways, for example [CLS] only:

instead of pooled_output = outputs[1],
use pooled_output = outputs[0][:,0,:]

This worked fine for me and predictions were satisfying.

2) There seems to be a problem with the optimization process ( I am still digging into this) which calls Adam optimizer from transformers library, or something is going on with the optimizer and my setup. Even, if using only the attention weights as inputs for the classification layer, the model shouldn't predict 0 for all labels. In addition when learning_rate equal or larger than 4e^-3 the model "breaks" and turns all predictions to 0.

[Jupiter79]

Okay so, some more tests I tried:

* Balance the data to remove class imbalance

* Run a classifier using FastText vs BERT-based

What I observed was that when I balanced the classes out (9 classes, so ~11% for each), the result using BERT was about 30%. Using FastText with imbalanced classes as well as using balanced classes yielded subpar results, 28.8% for the balanced case for example.

I also ran a classifier using NL BERT on the Dutch 110k book review dataset, used in this paper https://arxiv.org/abs/2001.06286. They show both a Dutch roBERTa model and the BERT model I am using. The BERT model scores 93% accuracy on this dataset.

I tried to gather as much of the parameters as used in the paper (max_seq_length set to 128, ran for 1 epoch, batch_size 4, learning rate 1e-5). The result I obtained is an accuracy of 79.3%, far, far lower than what they obtained in their paper. Surely my paramters might be off by a bit, but not this much....

Concluding:

* It seems that, despite papers showing otherwise, using pre-trained embeddings on Twitter data to predict emojis or emotions is not really possible in Dutch.

* I am unable to reproduce results shown in papers by a large margin.

I do wonder this though: DeepMoji seems to also learn the word embeddings based on their classification task. Does SimpleTransformers freeze the n-1th layer, thus only learning the weights for the classification task? Or does it propagate back to the actual embeddings too?

All in all, I don't think my quest for the truth has much to do with SimpleTransformers but more so my application thereof. Also - I am so far very much underwhelmed by transformers in general as living up to the papers' results seems far from reality (due to whatever case, even if just my wrong application thereof).

I am not sure if this can help, but I ran into similar problem with fast-bert library. Maybe changing the loss function could help in your case. Here is the link to their discussion: https://github.com/kaushaltrivedi/fast-bert/issues/52
Maybe similar solution could have application in SimpleTransformers as well.

[ErikTromp]

I haven't tried @Lysimachos suggestions yet but I did manage to get my way via a different route, albeit very specific to my problem at hand.

Instead of using transformers I decided to stick to the original of https://github.com/bfelbo/DeepMoji and use that instead. After rewriting the entire repo to work properly with Python 3 I ran my dataset through that model. Results were very successful and even better than their paper, though on my own dataset.

The most notable difference is that DeepMoji induces/learns the word vectors itself too. I also tried to seed it with FastText vectors to give it a head start but surprisingly that resulted in far worse outcomes.

Also - the chain-thaw finetuning method they use is pretty useful for doing transfer learning, as we are all trying to do here.

Note that while not a transformer model, you can use DeepMoji for multilabel classification as well (and for other tasks than just emotion prediction).

[ThilinaRajapakse]

outputs[1] is the last layer hidden-state of the classification token after being processed by a linear layer and the tanh activation function. Unless I'm mistaken, that is the same thing being extracted by
outputs[0][:,0,:].

[venkatasg]

I have run into the same issue using BertForSequenceClassification in code, and while I haven't used any of Simple Transformers library functions, I feel like what I did might be useful here too since simpletransformers is based on the transformers library. Firstly, @Lysimachos, the first issue you raise isn't relevant. As @ThilinaRajapakse says, outputs[1] is the last hidden-state of the CLS token processed by a linear layer and a tanh activation function. The source code here reflects this.

Your second suggestion however seems to be related. I tried experimenting with different learning rates, and batch sizes (don't go above 32). Initially what I tried was setting my own custom dropout probability, since I believed this was an overfitting issue, like so: model = BertForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2, hidden_dropout_prob=0.5).to(device). One or a combination of these changes made my model work - what I am not sure. I read in another thread(which I can't find now, will look again) about 'caching issues' - I wonder if this has something to do with this?

In any case, my recommendation is to try lower batch sizes and learning rates, and maybe use a custom dropout probability. I will investigate further and see what, if anything I had done to fix the issue on my end.

[ErikTromp]

Loss or dropout might be something looking into yes. The learning rates and batch sizes (1-8 since my GPU can only deal with 8GB) I already experimented with but I saw no impact.

[Lysimachos]

Greetings, I think I solved it - it is the learning rate.

First of all as @ThilinaRajapakse and @venkatasg pointed out about considering the inputs of the classifier as inappropriate is irreverent. I got different results when I changed this, probably by chance.

The learning rate that is applied by defaults is 4e-5 which is OK for fine-tuning transformer-based models but not that good for the classification layer. What I did and got nice predictions was to change the learning rate of the classification layer to 1e-3 while keeping 4e-5 for the transformer-based model. So far I have only tried with Bert models.

So I modified the train function of ClassificationModel from simpletransformers.classification.classification_model from this:

        no_decay = ["bias", "LayerNorm.weight"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
                "weight_decay": args["weight_decay"],
            },
            {
                "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
                "weight_decay": 0.0,
            }
        ]

to this (needs to add the new argument of learning_rate_classifier to global_args):

        no_decay = ["bias", "LayerNorm.weight"]
        classifier_parameters = ["classifier"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in model.named_parameters() if
                           (not any(nd in n for nd in no_decay) and not any(nd in n for nd in classifier_parameters))],
                "weight_decay": args["weight_decay"],
            },
            {
                "params": [p for n, p in model.named_parameters() if
                           (not any(nd in n for nd in no_decay) and any(nd in n for nd in classifier_parameters))],
                "weight_decay": args["weight_decay"],
                "lr": args["learning_rate_classifier"]
            },
            {
                "params": [p for n, p in model.named_parameters() if
                           (any(nd in n for nd in no_decay) and not any(nd in n for nd in classifier_parameters))],
                "weight_decay": 0.0,
            },
            {
                "params": [p for n, p in model.named_parameters() if
                           (any(nd in n for nd in no_decay) and any(nd in n for nd in classifier_parameters))],
                "weight_decay": 0.0
                "lr": args["learning_rate_classifier"]
            },
        ]

[venkatasg]

https://github.com/huggingface/transformers/issues/64 indicates that transformer models in general, are very sensitive to the learning rate and other hyperparameters used.

[ErikTromp]

I think you may be right. I tried the changes @Lysimachos suggested and got results that are a lot better than before, albeit still disappointing overall. At least I get actual predictions now
In my case, the "traditional" DeepMoji architecture yields results 2-3 times better.

I also have a hunch that this may have to do with me doing emotion/sentiment on social data here and I've observed more than once before that pretrained word vectors (w2v, fasttext and also BERT) trained on non-social data like wiki or CC do not capture sentiment well. For example, because "good" and "bad" pretty much always occur in the same context, their vectors are almost identical while being pretty much the most opposite possible for sentiment tasks. The big difference is DeepMoji co-trains the vectors from scratch while fitting the net.

[venkatasg]

I seem to be having the most issues with RoBERTa. If anyone has any tips on how to set up training and hyperparameters for RoBERTa please let me know.

[ThilinaRajapakse]

Maybe a little bit of language model fine-tuning will help you guys? That functionality has been added now.

[venkatasg]

@Lysimachos @ThilinaRajapakse What is the rationale behind setting weight decay to zero for LayerNorm and bias weights?

[ThilinaRajapakse]

That is the general practice. Weight decay is not applied to normalization layers and bias weights. My understanding is that it is unnecessary as those don't usually overfit.

[flozi00]

Same problem here, accuracy of 98% but in prediction only getting 0 for all labels.
Tried Albert, Roberta, Bert, distilbert

Edit: Problem solved after completely reinstalling and rebooting

[fcggamou]

Hi, just sharing that I'm facing the same obstacles described here.
I'm trying to fine-tune a multi-label classification model based on an Spanish BERT model which was fine-tuned using a corpus from EHRs.

The fine tuned model works very good for Masked Language Modelling in this domain, so now I'm trying to fine tune it to identify terms (e.g. diagnosis, procedures) from a medical standard (SNOMED). I already have a labelled dataset for this.

So I have 5000+ labels which is obviously a lot, but after multiple failed attempts I am now attempting this with only 10 labels and am still failing.

I'm trying to predict obvious one-word examples which are present in the training dataset but even the raw-output of the prediction yields a higher score for a different label. Seems it is always yielding the label which is the most frequent one in the dataset.

I just bumped into this thread so I have not tried the advice above yet. Thanks a lot in advance for any suggestion.

[ThilinaRajapakse]

Lowering the learning rate and/or the number of training epochs seems to be the best solution to prevent the model from breaking completely and predicting the same class.

[fcggamou]

Thanks @ThilinaRajapakse , I will give that a try. Although after giving this more though, it seems my problem is not exactly fit for a classification model. I just learned about the concept of named-entity linking (NEL), also known as named-entity recognition and disambiguation (NERD), which is exactly what I am trying to achieve. Sadly hugging face's transformers does not directly support this, but spaCy does.

[ThilinaRajapakse]

Both Simple Transformers (see here) and Hugging Face Transformers directly supports named entity recognition (NER).

[fcggamou]

Thanks @ThilinaRajapakse , I am aware of the NER support, I may actually use a transformers or simple transformers NER model to power the NEL model. But NEL is one step further from NER, as it tries to disambiguate the recognized entity.
E.g. a NER model would help me tag "high blood pressure" as a "disease", but I need it to point out what disease am I exactly talking about (i.e. point out the specific ID from a medical standard knowledge base). I just learned this kind of problem is called NEL.

[ThilinaRajapakse]

Got it!

[tutubalinaev]

Thanks @ThilinaRajapakse , I am aware of the NER support, I may actually use a transformers or simple transformers NER model to power the NEL model. But NEL is one step further from NER, as it tries to disambiguate the recognized entity.
E.g. a NER model would help me tag "high blood pressure" as a "disease", but I need it to point out what disease am I exactly talking about (i.e. point out the specific ID from a medical standard knowledge base). I just learned this kind of problem is called NEL.

I recommend utilizing a recent model from ACL 2020 instead of a classifier https://github.com/dmis-lab/BioSyn

[pmachin]

@ThilinaRajapakse,
@Lysimachos posts above relating a separate classification rate does increase my accuracy as well: https://github.com/ThilinaRajapakse/simpletransformers/issues/234#issuecomment-605877405
Would it be possible to integrate this insight into the next version?
Thanks for all your hard work!

[ThilinaRajapakse]

Yes, this can be added in a future release. Would you mind opening a feature request issue so that I don't forget about it? This thread is getting a little long.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[krannnn]

Greetings, I think I solved it - it is the learning rate.

First of all as @ThilinaRajapakse and @venkatasg pointed out about considering the inputs of the classifier as inappropriate is irreverent. I got different results when I changed this, probably by chance.

The learning rate that is applied by defaults is 4e-5 which is OK for fine-tuning transformer-based models but not that good for the classification layer. What I did and got nice predictions was to change the learning rate of the classification layer to 1e-3 while keeping 4e-5 for the transformer-based model. So far I have only tried with Bert models.

So I modified the train function of ClassificationModel from simpletransformers.classification.classification_model from this:

        no_decay = ["bias", "LayerNorm.weight"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
                "weight_decay": args["weight_decay"],
            },
            {
                "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
                "weight_decay": 0.0,
            }
        ]

to this (needs to add the new argument of learning_rate_classifier to global_args):

        no_decay = ["bias", "LayerNorm.weight"]
        classifier_parameters = ["classifier"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in model.named_parameters() if
                           (not any(nd in n for nd in no_decay) and not any(nd in n for nd in classifier_parameters))],
                "weight_decay": args["weight_decay"],
            },
            {
                "params": [p for n, p in model.named_parameters() if
                           (not any(nd in n for nd in no_decay) and any(nd in n for nd in classifier_parameters))],
                "weight_decay": args["weight_decay"],
                "lr": args["learning_rate_classifier"]
            },
            {
                "params": [p for n, p in model.named_parameters() if
                           (any(nd in n for nd in no_decay) and not any(nd in n for nd in classifier_parameters))],
                "weight_decay": 0.0,
            },
            {
                "params": [p for n, p in model.named_parameters() if
                           (any(nd in n for nd in no_decay) and any(nd in n for nd in classifier_parameters))],
                "weight_decay": 0.0
                "lr": args["learning_rate_classifier"]
            },
        ]

@Lysimachos @ThilinaRajapakse can you please tell me where to add this to simpletransformers code ? I'm doing multi-label classification and I think I'm facing a similar issue, but I don't know where to add this code to make it work. Thanks!

[krannnn]

@ThilinaRajapakse,
@Lysimachos posts above relating a separate classification rate does increase my accuracy as well: #234 (comment)
Would it be possible to integrate this insight into the next version?
Thanks for all your hard work!
@pmachin
Can you please share the changes you made to the code to make it work ?
Thanks

[ThilinaRajapakse]

You don't need to handle this manually anymore. Check docs here.