Farm: Training a language model in sagemaker with spot machines.

[GuillemGSubies]

Hello,

First of all, I'm new to AWS and I might be missing something, I'm sorry for that.

I'm trying to train a language model from scratch in AWS sagemaker using spot machines following this. This idea comes from this AWS summit talk where they say that you can save up to 70%.

However I have some doubts with respect to the checkpoints. First, what happens to the data when we start from a checkpoint? Does sagemaker know the data that has already been used or it just starts again from the beginning of the dataset?

I wonder if it is possible to see the code used to train the German BERT o at least having more details on how to train a language model from scratch using spot.

Thanks you and thanks for this great repo :)

[tholor]

Hey @GuillemGSubies ,

please have a look at this script which is a fully working example of running LM training on AWS sagemaker with checkpointing, DDP parallelization, AMP and streaming datasilo to not load all data into RAM.

For running this on SageMaker you would typically build a docker with this script and create a training job from there.
We, therefore, published also a prebuild "AWS SageMaker algorithm" that you can just use straight away. It's basically the above script + SageMaker configurations + a Docker image with GPU support. You can find it here:
https://aws.amazon.com/marketplace/pp/deepset-BERT-Training-from-Scratch/prodview-bpq22kfqxt376

First, what happens to the data when we start from a checkpoint? Does sagemaker know the data that has already been used or it just starts again from the beginning of the dataset?

We save the information about the last step in the checkpoint. When starting again we skip all data until there. This skipping can take some time as the implementation is currently not the most efficient, but it works well.

[GuillemGSubies]

Ok, thanks a lot for the fast response. I will study it and try it :blush:

[tholor]

Great. Closing this then for now. Feel free to re-open if further questions come up

[GuillemGSubies]

Hi again!

We are about to try the SageMaker algorithm, and after checking all the documentation, looks like the only model that can be used is BERT. Is there a way to use another base model (roBERTa, for example) with the SageMaker algorithm? Or we would have to tune your script and build our own infrastructure?

Thank you in advance

[tholor]

Yes, the published algo there is BERT only, mainly because the maintenance of more variants is too much overhead for us at the moment. However, you could rather easily modify the above-mentioned script + use a dockerfile like this to setup your own SageMaker algorithm for roberta