Gluon-ts: Trouble with hyper-parameters and additional features

[mariosantosprivate]

Hi, I'm having some trouble tuning my model (Deep AR), and deciding whether to add or remove some features.
This is because I've noticed training the model repeatedly on the same input data and with the same parameters will result in slightly different forecasts and therefore different error metrics. (I'm assuming this is due to the random nature of the sampling)

Given this, how would one go about performing hyper-parameter tuning, or adding features and checking if they're improving the performance? I considered training the model 5 times with each hyper-parameter and feature configuration and averaging the metrics, is that standard practice?

[kaijennissen]

@mariosantosprivate How big is the difference? For hyperparameter tuning I usually use only a single run since the difference in the error metric is often large enough to select a set of hyperparameters.

[mariosantosprivate]

@mariosantosprivate How big is the difference? For hyperparameter tuning I usually use only a single run since the difference in the error metric is often large enough to select a set of hyperparameters.

@kaijennissen I ran it twice with the same data and parameters and got these results:

1st run
MASE 0.357692
MAPE 0.139326
sMAPE 0.277678
MSIS 2.481089
RMSE 1,581.499704
NRMSE 0.260639
MRMSSE 0.152469
ND 0.168609

2nd run
MASE 0.442053
MAPE 0.172011
sMAPE 0.291773
MSIS 2.623137
RMSE 1,876.670623
NRMSE 0.309285
MRMSSE 0.213165
ND 0.209158

Do you think the differences are significant? I'm unsure about most metrics but the difference in MASE seems significant

Code:

train_ds = ListDataset([
    {
        FieldName.TARGET: target,
        FieldName.START: start,
        FieldName.FEAT_DYNAMIC_REAL: fdr,
        FieldName.FEAT_STATIC_CAT: fsc
    }
    for (target, start, fdr, fsc) in zip(train_target_values,
                                         dates,
                                         train_dynamic_features_list,
                                         stat_cat)
], freq="D")

test_ds = ListDataset([
    {
        FieldName.TARGET: target,
        FieldName.START: start,
        FieldName.FEAT_DYNAMIC_REAL: fdr,
        FieldName.FEAT_STATIC_CAT: fsc
    }
    for (target, start, fdr, fsc) in zip(test_target_values,
                                         dates,
                                         test_dynamic_features_list,
                                         stat_cat)
], freq="D")

deepAR_estimator = DeepAREstimator(
    prediction_length=prediction_length,
    context_length=2*prediction_length,
    freq="D",
    num_layers=5,
    num_cells=40,
    distr_output = NegativeBinomialOutput(),
    use_feat_dynamic_real=True,
    use_feat_static_cat=True,
    cardinality=stat_cat_cardinalities,
    dropout_rate=0.1,
    trainer=Trainer(
        learning_rate=1e-3,
        epochs=15,
        num_batches_per_epoch=500,
        batch_size=32
    )
)

[kaijennissen]

@mariosantosprivate How big is your dataset? I think just 15 epochs for training are not much.

[mariosantosprivate]

@kaijennissen the dataset consists of 934 time series each with 942 time points (daily sales)
The target values are the daily sales and I'm using 30 features (weather, promotional, holidays) as feat_dynamic_real, and 4 features as feat_static_cat (store_id, store_type, city etc..)

What would be your recommendations for this case?

[kaijennissen]

@mariosantosprivate What is your prediction_length? For daily sales I would try to use at least 7 days as context_length.
I usually use 200 epochs in combination with early_stopping (which is achived by supplying an additional dataset like this 'estimator.train(training_data=train_ds, validation_data=val_ds)). In addition I would increase thebatch_sizeand decrease thenum_batches_per_epoch`.

Of couse all these parameters can be fined tuned, but the early_stopping ensures that you stop training as soon as you are not getting better but also not to early, which I guess is the case with your current number pf epochs.

[mariosantosprivate]

@mariosantosprivate What is your prediction_length? For daily sales I would try to use at least 7 days as context_length.
I usually use 200 epochs in combination with early_stopping (which is achived by supplying an additional dataset like this 'estimator.train(training_data=train_ds, validation_data=val_ds)). In addition I would increase thebatch_sizeand decrease thenum_batches_per_epoch`.

Of couse all these parameters can be fined tuned, but the early_stopping ensures that you stop training as soon as you are not getting better but also not to early, which I guess is the case with your current number pf epochs.

@kaijennissen My prediction length is 48 days, so context length is 96 in this case. I thought that early stopping didn't need the validation set, as I've witnessed it stop before achieving the total number of epochs previously (at the time tried with 100 epochs, and training stopped by itself around epoch 80, no validation set supplied)

I'm already performing validation by not using the last prediction_length in training while tuning hyper-parameters, would it still make sense to use the validation_data you mentioned? And if so what data should that validation set contain?

[kaijennissen]

@mariosantosprivate Yes, early_stopping does not need a validation set but without the validation set, the model trains as long as the training loss decreases. Using a validation set training stops as soon as the loss on the validation set (and therefore on unseen data) does not decrease any further.

My recommendation would be to have three datasets (train, validation, test).
Where validation as well as test contains observations which were not used for training. F.e. 2018 as training set, the first half of 2019 as validation set and the second half of 2019 as test.

[mariosantosprivate]

@mariosantosprivate Yes, early_stopping does not need a validation set but without the validation set, the model trains as long as the training loss decreases. Using a validation set training stops as soon as the loss on the validation set (and therefore on unseen data) does not decrease any further.

My recommendation would be to have three datasets (train, validation, test).
Where validation as well as test contains observations which were not used for training. F.e. 2018 as training set, the first half of 2019 as validation set and the second half of 2019 as test.

Thank you very much I will try that!

Just to close the original question and make sure I understood correctly, even though there is a random component to sampling, given enough epochs there should be no significant differences in metrics when training a model with the same input and parameters?

[kaijennissen]

@mariosantosprivate Given enough data, I would hope that a global minimum is reached and therefore results from different runs are comparable. See for example my results (code below) from two runs with DeepAR on the electricity dataset.

Run 1:

• MASE: 0.8078506007638229
• MAPE: 0.09673023279096417
• sMAPE: 0.11191548997159273

Run 2:

• MASE: 0.8161786940956743
• MAPE: 0.09720666821452344
• sMAPE: 0.11201153361129423

from gluonts.dataset.repository.datasets import get_dataset
from gluonts.evaluation import Evaluator
from gluonts.evaluation.backtest import make_evaluation_predictions
from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer
from gluonts.mx.distribution import StudentTOutput


dataset = get_dataset("electricity")
metadata = dataset.metadata

deepar = DeepAREstimator(
    freq=metadata.freq,
    prediction_length=72,
    context_length=168,
    num_layers=1,
    num_cells=50,
    cell_type="lstm",
    dropout_rate=0.1,
    scaling=True,
    distr_output=StudentTOutput(),
    trainer=Trainer(),
)

def main(estimator):

    predictor = estimator.train(
        training_data=dataset.train, num_workers=8)

    QUANTILES = [0.1, 0.9]
    evaluator = Evaluator(
        quantiles=QUANTILES,
        num_workers=8
    )

    forecast_it, ts_it = make_evaluation_predictions(
        dataset=dataset.test,
        num_samples=200,
        predictor=predictor,
    )

    forecasts = list(forecast_it)
    tss = list(ts_it)
    agg_metrics, item_metrics = evaluator(tss, forecasts)

    metrics = [
        "MASE",
        "MAPE",
        "sMAPE",
    ] + \
        [
        f"QuantileLoss[{QUANTILES[i]}]" for i in range(len(QUANTILES))] + \
        [
        f"Coverage[{QUANTILES[i]}]" for i in range(len(QUANTILES))
    ]

    for metric in metrics:
        print(f"{metric}: {agg_metrics[metric]}")


if __name__ == "__main__":

    main(deepar)
    main(deepar)

[mariosantosprivate]

Solved, thank you!

[bugsuse]

I'm new to gluton, and the estimator's train is not like pytorch, the training and validation phase are separate. such as,

    predictor = estimator.train(
        training_data=dataset.train, num_workers=8)

    QUANTILES = [0.1, 0.9]
    evaluator = Evaluator(
        quantiles=QUANTILES,
        num_workers=8
    )

    forecast_it, ts_it = make_evaluation_predictions(
        dataset=dataset.test,
        num_samples=200,
        predictor=predictor,
    )

It's not like pytorch,

for epoch in range(num_epoches):
    train()
    eval()

I want to know how to apply early_stopping to estimator? Thanks so much! @kaijennissen

[mariosantosprivate]

I'm new to gluton, and the estimator's train is not like pytorch, the training and validation phase are separate. such as,

    predictor = estimator.train(
        training_data=dataset.train, num_workers=8)

    QUANTILES = [0.1, 0.9]
    evaluator = Evaluator(
        quantiles=QUANTILES,
        num_workers=8
    )

    forecast_it, ts_it = make_evaluation_predictions(
        dataset=dataset.test,
        num_samples=200,
        predictor=predictor,
    )

It's not like pytorch,

for epoch in range(num_epoches):
    train()
    eval()

I want to know how to apply early_stopping to estimator? Thanks so much! @kaijennissen

Hello @bugsuse, if I understood correctly, as @kaijennissen mentioned you supply the estimator with a validation dataset, as follows:

(...) I usually use 200 epochs in combination with early_stopping (which is achived by supplying an additional dataset like this 'estimator.train(training_data=train_ds, validation_data=val_ds)`) (...).

So for your example:

predictor = estimator.train(training_data=dataset.train, validation_data = dataset.val, num_workers=8)

Say your dataset.train has 90 days of data, split it so that dataset.train has 70 days and dataset.val has 90 days for example (including the 70 days in dataset.train)

I hope that helps!

[bugsuse]

I'm new to gluton, and the estimator's train is not like pytorch, the training and validation phase are separate. such as,

    predictor = estimator.train(
        training_data=dataset.train, num_workers=8)

    QUANTILES = [0.1, 0.9]
    evaluator = Evaluator(
        quantiles=QUANTILES,
        num_workers=8
    )

    forecast_it, ts_it = make_evaluation_predictions(
        dataset=dataset.test,
        num_samples=200,
        predictor=predictor,
    )

It's not like pytorch,

for epoch in range(num_epoches):
    train()
    eval()

I want to know how to apply early_stopping to estimator? Thanks so much! @kaijennissen

Hello @bugsuse, if I understood correctly, as @kaijennissen mentioned you supply the estimator with a validation dataset, as follows:

(...) I usually use 200 epochs in combination with early_stopping (which is achived by supplying an additional dataset like this 'estimator.train(training_data=train_ds, validation_data=val_ds)`) (...).

So for your example:

predictor = estimator.train(training_data=dataset.train, validation_data = dataset.val, num_workers=8)

Say your dataset.train has 90 days of data, split it so that dataset.train has 70 days and dataset.val has 90 days for example (including the 70 days in dataset.train)

I hope that helps!

Sorry for I ignored the information! Thanks a lot! @mariosantosprivate