Simpletransformers: Early stopping

[kinoute]

Hi,

Thanks for this great library. It works nicely. Is there any chance you could add a "early stopping" feature? At the moment I struggle to monitor my training to avoid overfitting with only the running loss being displayed.

Thanks!

[ThilinaRajapakse]

Yeah, I'll see what I can do.

You can also use tensorboard to get more details on the training process. The progress is logged to the runs/ directory.

[kinoute]

Thanks for the answer. Regarding Tensorboard, It works indeed with your library.

Though, when we train a model (and eval during training as well), is there a way to display the validation loss? I only see a "loss" graph in tensor board which I guess is the training loss. I saw some "eval_tp/eval_fp" stuff but I don't see anything related to the eval_loss which, according to the eval_model function, should be returned. Am I missing something?

Here is a screenshot:

Thanks!

[ThilinaRajapakse]

You are right, the loss graph is showing the training loss.

The eval_model function doesn't return the loss right now. It's returning the mcc score, and TP, TN, FP, FN counts. Those are the values getting sent to Tensorboard as well.

I'm adding the loss in the next update but you can get it right now by making a small change to the evaluate function.

You have to add a line to the evaluate function in the classification/classification_models.py file.

        result, wrong = self.compute_metrics(preds, out_label_ids, eval_examples, **kwargs)
        result['eval_loss'] = eval_loss  # This is the line to add.
        results.update(result)

[kinoute]

Cheers for adding this so quickly in 0.7.0. It works perfectly, both appearing during training in jupyter and TensorBoard. There is just a little typo but no big deal:

Can't wait for the early_stopping feature. You're doing an awesome job with this library. Keep it up.

[ThilinaRajapakse]

Whoops, guess 'eval_' is being prepended for all the metrics.

Thank you for the encouragement!

[kinoute]

In the meantime, it would be great if the eval results of every validation during training is saved in the checkpoint* folders. At the moment, you can see the eval results during the training on your notebook for example, while it is happening. But at the end of the training, the eval_results.txt saved in the outputs folder is only about the latest.

I'm trying to find the best way to re-train my models in an automatic way and take the best model with the best validation loss. Before the early stopping feature appears, would it be possible to have a eval results.txt for every checkpoint/in every checkpoint folder? At least it could help to find the best model programmatically. I know there is TensorBoard, and some files in the runs/ folder but I'm not sure I can exploit it programmatically to get the checkpoint with the best validation loss.

Thank you!

[ThilinaRajapakse]

I updated the evaluation during training procedure. You can now specify the number of steps at which evaluation will be done using evaluate_during_training_steps. The model checkpoint and the evaluation results will be saved.

This is separate from logging_steps and save_steps, which I think gives more flexibility in how you want the training to go (albeit at the cost of a little more complexity).

Sorry this took so long!

[kinoute]

Thank you so much! I just tested it and indeed, I have a eval_results.txt file in every checkpoint folder. That will make my life easier to programmatically pick the best checkpoint instead of decoding the tensorboard events file.

I see two things that could make things even better:

• Save the results as a .csv file instead of a txt file. More interestingly, save the result of every checkpoint at the end in a csv file that sums up the training. Like:

| checkpoint | accuracy | validation_loss |
| ---------- | -------- | ---- |
| 50 | 0.89 | 0.88 |
| 100 | 0.932 | 0.50 |
| 150 | 0.95 | 0.32 |

• Be able to add more metrics to the train_model method. At the moment, we can add additional metrics to eval_model which we call at the end of the training, but it would be great if these custom metrics could be also used for the evaluation during training.

Like:

from sklearn.metrics import (
    cohen_kappa_score, 
    confusion_matrix, 
    classification_report, 
    balanced_accuracy_score, 
    confusion_matrix
)

model.train_model(train_dataset, eval_df=val_dataset, confusion=confusion_matrix, etc..)

[ThilinaRajapakse]

Excellent ideas!

I'll add them soon.

[ThilinaRajapakse]

I've implemented your suggestions. I'm also looking into early stopping as well as additional visualization support.

[kinoute]

Once again, great work! It definitely makes things easier. You could also add the training loss in the CSV file to help people create visualizations like training loss vs. validation loss (and see the moment it overfits).

I don't know if I'm blind but I recently discovered that there are also "epoch-X" folders created during training. I'm not sure if it was added recently or it has always been here but it should be taken in consideration for the summary since, if I'm not wrong, at the end of an epoch, an evaluation is done as well. I just did a test with 2 epochs, an evaluation every 50 steps and the validation of the 2 epochs are not present in the CSV file (I mean the eval_results.txt data present in the "epoch-x" folders). Since these two "additional" evaluations can be seen as checkpoints, I think they should be included.

I have a dumb question by the way related to checkpoints: if, out of 3 epochs and a checkpoint every 50 steps, the best validation_loss was on a checkpoint before reaching the first epoch (i.e seeing all my training data). Would it be OK to use it or I should _at least_ wait for a full epoch to be done and then pick the best checkpoint afterwards, if any?

[ThilinaRajapakse]

I had a feeling I was missing something. Training loss should have been added indeed.

I was debating whether or not to add the metrics from the end of epoch evaluations. I decided to leave it out to keep the number of steps between the checkpoints constant. I feel adding them would make things a little messy. If necessary, one could choose a evaluation step value so that it coincides with the end of the epoch.

There are no dumb questions!
It would depend heavily on the situation. In general, I would say that it is ok to pick the best checkpoint regardless of whether or not an epoch has been completed. There are some caveats I'd suggest keeping in mind though.

• The best checkpoint coming very early in the first epoch might indicate an issue with either the training data or validation data.
• Might be a good idea to investigate why the validation scores drop in the first epoch itself. Learning rate might be too low or training and validation datasets might be too different from each other, for example.

If you want to be really scientific, you could test the best checkpoint on the untrained training data and see whether it differs significantly from the validation performance.

[kinoute]

Thanks for your answer!

About the epochs evaluation data, I understand your point. Right now, it might look confusing. Maybe later we can think about a way to integrate them correctly. Something like:

Epoch | Checkpoint | eval_loss
-- | -- | --
  | 50 | 0.63
  | 100 | 0.25
1 |   | 0.18
  | 150 | 0.14
  | 200 | 0.09

[kinoute]

Thank you for your last additions, you rock!

[ThilinaRajapakse]

You are welcome and thank you!
I had a feeling you'd enjoy this update. 😉 I was going to tag it here after updating the readme.

[ThilinaRajapakse]

Early stopping has been added for classification tasks (finally!). It seems to be working fine in my tests, so if nobody finds any issues, I'll extend it to NER and QA as well.

[kinoute]

@ThilinaRajapakse Awesome work! You add so many things that quick that I have to rebuild my docker images and change my code every day ahah.

I tried the function quickly a few moments ago, looks good! Will try more tomorrow.

[ThilinaRajapakse]

Haha! That's my evil plan. 😂

[kinoute]

@ThilinaRajapakse I think I found an issue but I will write more about it tomorrow (and also maybe create a PR for just more logging stuff for early stopping).

TLDR; If the "best checkpoint" according to eval_loss is a "epoch" evaluation (not a regular checkpoint in between), the best model directory doesn't seem to save it.

[ThilinaRajapakse]

You are right! I'll fix this tomorrow (or merge if you do).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[lockingkao]

May I asked how to visualize training progress by tensorboard or visdom?
I don't know how to visualize the training loss and evaluation loss
like keras "history" function!
Thanks a lot!

[ThilinaRajapakse]

The easiest way to do it would be to use the integrated W&B support. You can see it here.

[lockingkao]

The easiest way to do it would be to use the integrated W&B support. You can see it here.

Thank you so much!
May I ask when I use the simple transformer,does the evaluation data evaluate the same time?

I usd train_test_split to split the data
train_df, eval_df = train_test_split(df, test_size=0.25)

result, model_outputs, wrong_predictions = model.eval_model(eval_df)
but I can't find the eval_loss on the graph!
thank you very much!

[ThilinaRajapakse]

For that, you need to do evaluation_during_training=True. That also means you have to provide the eval_df to the train_model() function. (You can find all the options here, look for evaluate_during_X options)

For example,

from simpletransformers.classification import ClassificationModel, ClassificationArgs


model_args = ClassificationArgs()
model_args.evaluation_during_training = True
model_args.evaluation_during_training_steps = 100

model = ClassificationModel("bert", "bert-base-cased", args=model_args)

train_df, eval_df = train_test_split(df, test_size=0.25)

model.train_model(train_df, eval_df=eval_df)

There is also a small tutorial here.

[lockingkao]

For that, you need to do evaluation_during_training=True. That also means you have to provide the eval_df to the train_model() function. (You can find all the options here, look for evaluate_during_X options)

For example,

from simpletransformers.classification import ClassificationModel, ClassificationArgs


model_args = ClassificationArgs()
model_args.evaluation_during_training = True
model_args.evaluation_during_training_steps = 100

model = ClassificationModel("bert", "bert-base-cased", args=model_args)

train_df, eval_df = train_test_split(df, test_size=0.25)

model.train_model(train_df, eval_df=eval_df)

There is also a small tutorial here.

Thank you so much!
But I encounter this ERROR:

ValueError: Expected input batch_size (32) to match target batch_size (288).

My arguments:

train_args={
'evaluate_during_training': True,
'num_train_epochs': 300,
'logging_steps': 10,
'evaluate_during_training_steps': 10,
'save_eval_checkpoints': False,
'train_batch_size': 32,
'eval_batch_size': 32,
'reprocess_input_data': True,
'gradient_accumulation_steps':16,
'learning_rate': 3e-5,
'overwrite_output_dir': True,
'use_multiprocessing':False,
'TOKENIZERS_PARALLELISM':False,
'wandb_project': "BERT-visualization-demo",
'max_seq_length': 512}

[ThilinaRajapakse]

I'm not sure what that's about. Can you share your full training script?

[lockingkao]

This is my code!But it still not working because of that following error.

ValueError: Expected input batch_size (32) to match target batch_size (288).

[lockingkao]

Do you have any answers?

[ThilinaRajapakse]

Nothing really jumps out in the code. Can you post the full stack trace of the error? I need to see which line is causing the error, that might provide a clue.

It might be best to start a new issue for this, btw. This is likely to get missed as the issue is closed.

_Please copy and paste the error/code. You can put it inside backticks for better formatting._

[tommasodelorenzo]

Hi, stupid question: I did not understand if and how I can use custom metrics (such as skLearn ones) as early stopping metric.
I naively tried
model_args.early_stopping_metric = sklearn.metrics.whatever_metric
but did not work.

KeyErrorTraceback (most recent call last)
<ipython-input-16-9ea097139f0a> in <module>
----> 1 model.train_model(train_df = df_train, eval_df = df_val)

/opt/conda/lib/python3.8/site-packages/simpletransformers/classification/classification_model.py in train_model(self, train_df, multi_label, output_dir, show_running_loss, args, eval_df, verbose, **kwargs)
    461         os.makedirs(output_dir, exist_ok=True)
    462 
--> 463         global_step, training_details = self.train(
    464             train_dataloader,
    465             output_dir,

/opt/conda/lib/python3.8/site-packages/simpletransformers/classification/classification_model.py in train(self, train_dataloader, output_dir, multi_label, show_running_loss, eval_df, verbose, **kwargs)
    789 
    790                         if not best_eval_metric:
--> 791                             best_eval_metric = results[args.early_stopping_metric]
    792                             self.save_model(args.best_model_dir, optimizer, scheduler, model=model, results=results)
    793                         if best_eval_metric and args.early_stopping_metric_minimize:

KeyError: <function balanced_accuracy_score at 0x7f3aa138ba60>

Thanks for the help

[ThilinaRajapakse]

https://simpletransformers.ai/docs/tips-and-tricks/#additional-evaluation-metrics