Simpletransformers: How to load checkpoints? and approximate run time for Multi-label classification example.

[legolego]

Hello,
Thank you for the library, I'm going over the Multi-label classification post of yours (https://towardsdatascience.com/multi-label-classification-using-bert-roberta-xlnet-xlm-and-distilbert-with-simple-transformers-b3e0cda12ce5). I have two questions really...
I ran the code in the post and I let it go overnight on my laptop, (Intel i7 and Nvidia 1060, 24gb ram) and it didn't finish, so I stopped it.... how long should that example take to run?
I also noticed that there were checkpoints in the model_output folder... up to checkpoint-8000. What's the right way to load this so I can continue from this spot? Is there a 'last' checkpoint? How much longer do you think it might take to finish? Thank you very much!

[ThilinaRajapakse]

It's hard to give an estimate on the time but I'm not surprised it didn't complete overnight. It took a few hours to run on a RTX 2080. The progrt bars will tell you how long is left when you are running the training.

model = MultiLabelClassificationModel('roberta', 'outputs/checkpoint-8000', num_labels=6}

This will load the model from the checkpoint. Make sure you pass in the same args dictionary as when you started training.

[legolego]

Thank you for replying! :)
I continued the training with the the checkpoint loaded like you mentioned, but a new output directory because I wasn't sure about the overwrite flag. My new output directory now has checkpoint-2000 and 4000 in it, are those just a continuation from where it stopped at the previous 8000 checkpoint? Or did it start over? Would it be safe to set the overwrite flag to True in this case?
Also, I'm not seeing any progress bars in PyCharm, but it seems to be working. I ran your other minimal multi-label example (https://github.com/ThilinaRajapakse/simpletransformers#minimal-start-for-multilabel-classification) and had to set 'use_multiprocessing': False in model= for it to work. It was stuck in a loop of some sort with errors, but afterwards ran in about 1 second with progress bars. Would it make sense to set this flag here too? How much would it slow down? Or should I just wait to wait to see what happens here? I'm not getting errors now, just seeing new 'Running loss: xxxx' strings added in the output window.

[ThilinaRajapakse]

No problem!
Those are a continuation from the checkpoint so technically that should be 10000 and 12000 but the file names are just built from the current training step so the number itself resets if you rerun the training.
Setting overwrite output directory will overwrite any files that happen to have the same name as whatever is going to be saved. For example, the new checkpoint-2000 would overwrite the previous one.
Not sure why multiprocessing would throw errors. You wouldn't happen to be on Windows, would you? 🤔
Anyway, multiprocessing will only affect the conversion of text to features which is only done once at the beginning of training. How much slowdown you'll see will depend on how much data you have. By default, it uses number of cores - 2, so you can expect close to a 6x speedup with an 8-core CPU for the conversion part with multiprocessing.
The running loss is the current training loss so that means your model is training. As long as it isn't jumping around and is decreasing gradually, I suggest just letting it run.

[legolego]

Yes, I'm on Windows 10, PyCharm 2019.2, python 3.7. Do you know about how many training steps it takes for the toxic comment example to finish? I can post the error I was getting in the minimal example after this one finishes if you'd like, it just gave me a different memory error when I tried running it concurrently with the toxic comment example.

[ThilinaRajapakse]

Windows tends to be weird with multiprocessing so that's probably causing the error there. Might also be why the progress bars aren't showing up.

I can't remember the number of steps off the top of my head but you can calculate it if you like. It will be the number of samples in the training data, divided by the train_batch_size, multiplied by num_train_epochs.

Yeah, a 1060 probably won't have enough VRAM to run both at the same time.

[legolego]

Ok, it looks like I'm going to be here a while then with it running like it is :/
number of samples in the training data, divided by the train_batch_size, multiplied by num_train_epochs = (159571 rows/ 2 train_batch_size) * 3 num_train_epochs = 239356.5
I saw CUDA memory errors when I tried running that minimal example, but is there a way to make sure I'm using the video card? I do have 6gb of memory if that helps. I'm a little concerned because the Performance tab in Windows Task Manager show the GPU at 2% utilization.... do you know if there's a better way to confirm that?

[ThilinaRajapakse]

You can probably handle a batch size of 8 with 6 GB of memory. That should speed up the training.

If you are getting CUDA errors, that means you are using the GPU. Also, unless you manually set use_cuda=False, Simple Transformers will use CUDA. If you try to run without setting it to False when CUDA is _not_ available, it will error out rather than silently switching to CPU precisely because of this concern!

The Windows task manager doesn't give an accurate reading for Pytorch from what I know. You can use GPU-Z. It's accurate, gives a lot more details, and is far cooler looking!

[legolego]

Ok, that's good to know, thank you! :) GPU-Z shows just over 90% utilization and Memory Used is 5500MB, I feel better about that :) If I were to stop this run and start again from the most recent checkpoint, but with a batch size of 8, would that mess anything up?

[ThilinaRajapakse]

90% utilization and Memory Used is 5500MB

In that case, you are probably at the limit of your GPU's capabilities. Stopping and changing the batch size probably won't mess things up (you might get a temporary spike in loss) but I don't think you'll be able to increase the batchsize without getting memory errors. If you are only running this to check things out, it's probably safe to stop at 1 training epoch rather than 3. The model performance will be slightly lower but should still be acceptable.

[legolego]

I'll reboot after the next checkpoint is saved and try to bump it up and see what happens, maybe something else is using the video memory. Would it be safe to change the number of training epochs in the middle like this too, or is this value in the arg dictionary ok to change at this time?

[ThilinaRajapakse]

You can change it. If you restart with it set to 1, it will do one more training epoch.

[legolego]

Ok, does that mean 2 training epochs total then, if I'm only part way through the first?

[ThilinaRajapakse]

It would mean; whatever training has already been done + another training epoch.

The reason is that when you restart training from a checkpoint, there is no record of how much training has already been done. You are getting the benefits of the previous training (namely the weights of the model at the time the checkpoint was created) but as far as the training loop is concerned, you are starting from the beginning. Basically, the difference is that you have a model with random weights when starting from scratch, and you have a model with trained weights when you start from a checkpoint.

[legolego]

Ok, so could I make it a 0 to have it finish sooner, on this epoch? Thank you so much for the good answers to such basic questions!

[ThilinaRajapakse]

Changing it will have no effect unless you restart training. Restarting with it set to 0 will mean that no further training will be done.

Instead of that, you can simply do evaluation/prediction by using the last model checkpoint. For example, if your last checkpoint was 10000;

# Make sure you are using the correct model type. I am using roberta as an example. Same for other args. You only need to change the model_name, i.e 'roberta-base' to 'outputs/checkpoint-10000'
model = MultiLabelClassificationModel('roberta', '<checkpoint_directory>/checkpoint-10000', num_labels=6, args={<same args as before>})

# Now you can evaluate with the checkpoint
result, model_outputs, wrong_predictions = model.eval_model(eval_df)
print(result)
print(model_outputs)

You are welcome!

[legolego]

Ok, that helps me a lot! :)

[ThilinaRajapakse]

Great!

[legolego]

Ok, so I let it run, setting 'num_train_epochs': 1 and it stopped nicely, after 20500 steps total, and it seems like much too soon. I don't see any errors. Is there something I should look for to see why it stopped so early? I was expecting ~160k steps.

[ankurgoel256]

can you tell me how can I use early stopping with multilabel simpletransformer model

[ThilinaRajapakse]

Early stopping is not supported natively atm, but I do want to add it soon. #38

[parrisr]

It would mean; whatever training has already been done + another training epoch.

The reason is that when you restart training from a checkpoint, there is no record of how much training has already been done. You are getting the benefits of the previous training (namely the weights of the model at the time the checkpoint was created) but as far as the training loop is concerned, you are starting from the beginning. Basically, the difference is that you have a model with random weights when starting from scratch, and you have a model with trained weights when you start from a checkpoint.

Just to confirm: If I trained something over 5 epochs, and I like the results but I want to see if they would be better with 10 epochs instead, do I have to start over like this:

model = ClassificationModel('roberta', 'roberta-base', args={'num_train_epochs': 10})

Or can I just add another 5 epochs to the 5 I've already done like this:

model = ClassificationModel('roberta', 'outputs-5-epochs/', args={'num_train_epochs': 5})

I understand there may be differences due to randomness, but in theory are they equivalent or is adding another 5 to the existing 5 inferior in some way?

[djstrong]

@ThilinaRajapakse continuing training is not working properly. It loads the model, but e.g. learning rate is not loaded and warmup steps are performed again (I see it in W&B learning rate chart).

edit: OK, you have written that earlier.

[djstrong]

Optimizer and scheduler are not loaded (in ClassificationModel) like in: https://github.com/huggingface/transformers/blob/601ac5b1dc1438f00d09696588f2deb0f045ae3b/examples/ner/run_ner.py#L99-L104
They are also not saved in checkpoints so the training can not be continued.

[djstrong]

@parrisr to continue training you must know how learning rate (LR) is changing and set it to desired value (usually less than previous LR). If you trained successfully for 5 epochs then at the end LR is near 0. If you trained for 10 epochs and stopped at 5 then LR has some value (e.g. half of original LR). By setting number of epochs you define how slow LR is decreasing.

model = ClassificationModel('roberta', 'roberta-base', args={'num_train_epochs': 5, 'learning_rate':4e-5})
model = ClassificationModel('roberta', 'outputs-5-epochs/', args={'num_train_epochs': 5, 'learning_rate':2e-5})

[ThilinaRajapakse]

Yeah, this was only added in the last update. Classification, NER, QnA, and Language Modeling models should now resume training properly.

[jeugregg]

I cannot continue training with QnA model.
The learning rate is not the last one.
So it is not possible without update learning rate in args as said above ?

My tests :
After a first training of 3 epochs with args as arguments for model, I tried that :

model = QuestionAnsweringModel('distilbert', '\outputs\checkpoint-1887-epoch-3')
model.train_model(train_data, args=args)

or

model = QuestionAnsweringModel('distilbert', '\outputs\checkpoint-1887-epoch-3', args=args)
model.train_model(train_data, args=args)

or

model = QuestionAnsweringModel('distilbert', '\outputs\checkpoint-1887-epoch-3', args=args)
model.train_model(train_data)

The model training is not resume, at least for learning rate.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[darthgera123]

its not loading the checkpoints

[sarapapi]

I cannot continue training with QnA model.
The learning rate is not the last one.
So it is not possible without update learning rate in args as said above ?

My tests :
After a first training of 3 epochs with args as arguments for model, I tried that :

model = QuestionAnsweringModel('distilbert', '\outputs\checkpoint-1887-epoch-3')
model.train_model(train_data, args=args)

or

model = QuestionAnsweringModel('distilbert', '\outputs\checkpoint-1887-epoch-3', args=args)
model.train_model(train_data, args=args)

or

model = QuestionAnsweringModel('distilbert', '\outputs\checkpoint-1887-epoch-3', args=args)
model.train_model(train_data)

The model training is not resume, at least for learning rate.

Have you found any solution?

[jeugregg]

no. I use always the same learning rate at every resume. I don't use warmup ratio (=0).
But it ends for me because my problem is too special to be use with BERT. I cannot tell you more.

[sarapapi]

no. I use always the same learning rate at every resume. I don't use warmup ratio (=0).
But it ends for me because my problem is too special to be use with BERT. I cannot tell you more.

Thank you anyway!