Gluon-ts: Incremental Training

[Hari7696]

Description

• As we are working on Timeseries, new data keeps flowing in.
• So every time, when new data comes we have to train the entire model again which is costly and time consuming.
• INCREMENTAL TRAINING would be great to deal with this problem.

References

• Some thing similar to transfer learning which we can do with tensorflow and other frame works
• list known implementations : Incremental training in Amazon sage maker(as of now it supports object detection and image classification algorithms only)

[jaheba]

Thanks for the suggestion.

I think there are two main challenges:

• Technical: We need an API for this. Not all models can be re-trained, or it should be at least the choice of the author to add incremental learning or not.

• Scientific: A possible problem I see is that the model could overfit on the new data. Thus, we need a lot of experimentation to investigate the impact of incremental learning on the overall accuracy.

I guess, we could start with implementing an API which then enables us try things out. However, we should be careful to not advertise this API as stating that we can do transfer learning in forecasting.

[geoalgo]

The technical challenge for the API should be reasonable, we can just pickle the result of a training.

    train_output = estimator.train_model(dataset.train)
    # then 
    # 1) pickle train_output
    # 2) deserialize it and train another time

[live-high]

Any more progress?

[mbohlkeschneider]

Hi @live-high,

thank you for your interest!

Incremental training is currently not on our roadmap and there has been no activity on this from our side.

[live-high]

@mbohlkeschneider Thanks to reply!
BTW, is the incremental training not so important for time series prediction?

[mbohlkeschneider]

That depends on the use case. For very large data and/or data that is coming in with high velocity, it would be quite useful. Also, to reduce variation in the model between re-trains. I think work in this direction has quite some value. However, just re-training the model with new data works just fine in many cases.

[live-high]

You're right, it depends.
In our project, data stream real-time produced and we want our prediction model could fine-tune from a pre-trained model.
Thanks for replying, maybe we'll try some workarounds later.

[cesans]

I've come across with this issue as well. Looking at the GluonEstimator.trainer() code it returns only the predictor:

    def train(...):
        return self.train_model(
            training_data, validation_data, num_workers, num_prefetch, **kwargs
        ).predictor

But that TrainOutput object also contains the trained_net, I think it is possible to keep that object and allow to continue the training with minor API changes. Something like this:

    def train(..., continue_training=False):
        self.trained_model =  self.train_model(..., continue_training, ...)
        return self.trained_model.predictor

    def train_model(..., continue_training=False):
        ...
        if continue_training:
            trained_net = self.trained_model
        else:
            with self.trainer.ctx:
                trained_net = self.create_training_network()
        ...

Would this make sense? I am not sure if there is any model not using the GluonEstimator train/model_train but looking at the DeepAR model this seems to work.

[cpyy103]

When I use DeepAREstimator on gpu, I add some the code in GluonEstimator.train_model() to load Predictor's params to trained_net

And it seems to work, although I get warning like UserWarning: Parameter 'deepartrainingnetwork2_None_distr_mu_weight' is already initialized, ignoring. Set force_reinit=True to re-initialize. v.initialize(None, ctx, init, force_reinit=force_reinit)

Cause in Trainer, the net will initialize, and the parameter force_reinit defaults to False. I think it's ok.

Is the above right?

        with self.trainer.ctx:
            trained_net = self.create_training_network()

        model_path = ...
        if continue_train and os.listdir(model_path):
            predictor_deserialized = RepresentableBlockPredictor.deserialize(Path(model_path))
            net = predictor_deserialized.prediction_net
            trained_net.initialize()
            copy_parameters(net, trained_net)
            trained_net.collect_params().reset_ctx(self.trainer.ctx)