Gluon-ts: Predictions are way too high when modeling an intermittent count series with DeepAR and NegBin distribution?

[SkanderHn]

I'm trying to model a simulated series of weekly seasonal intermittent sales, with values between 0 and 4. I generated 5 years of simulated data:

I trained a DeepAR model with the output distribution set to Negative Binomial (all other settings were the default settings), on 3 years, and generated predictions for the next two. I got the following results (plotting the [70.0, 80.0, 95.0] predictions intervals):

Increasing number of training epochs doesn't change anything, the loss falls to its lowest value around the 8th to 10th epoch and hovers more or less around there, whether I train for 10 or 100 epochs.
I thought training on 3 years and testing on 2 might be too ambitious, so I tried 4y/1y split instead, and the results got much worse - and downright strange - this time with values climbing into the 100s, even though the largest historical value the series ever reaches is 4 (I'm using the same input series, but is seems flat now because the scale is completely skewed by how large the predictions are):

I'm wondering if I am doing anything wrong? Are there any special settings for DeepAR when applied to intermittent series?

For comparison, the DeepAREstimator worked pretty well out of the box for more traditional series (using Student's distribution), for example:

Details:

Train data:
[{'start': Timestamp('2014-01-05 00:00:00', freq='W-SUN'),
'target': array([1., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 1.,
1., 1., 1., 1., 2., 0., 0., 1., 2., 2., 1., 4., 1., 2., 1., 0., 0.,
2., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0., 0., 0., 1., 2., 1., 2.,
0., 1., 1., 2., 3., 2., 2., 1., 1., 3., 4., 1., 1., 0., 0., 3., 0.,
0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 1., 1., 0., 1., 0., 0., 0., 1., 1., 0., 2., 1., 1., 0., 1., 0.,
1., 2., 2., 1., 2., 3., 3., 1., 2., 2., 0., 0., 2., 0., 3., 0., 1.,
2., 0., 1., 1.], dtype=float32),
'source': SourceContext(source='list_data', row=1)}]

Test data:
{'start': Timestamp('2017-01-08 00:00:00', freq='W-SUN'), 'target': array([2., 1., 2., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.,
0., 0., 1., 0., 1., 0., 0., 1., 0., 1., 0., 1., 2., 3., 1., 0., 3.,
2., 1., 0., 0., 2., 2., 2., 1., 0., 2., 0., 2., 2., 1., 0., 1., 0.,
0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 2., 0.,
0., 4., 1., 2., 2., 1., 3., 1., 2., 1., 2., 1., 2., 3., 3., 1., 2.,
0.], dtype=float32), 'source': SourceContext(source='list_data', row=1)}

Estimator used:

estimator = DeepAREstimator(freq="W", prediction_length=105, trainer=Trainer(epochs=10),distr_output=NegativeBinomialOutput()) predictor = estimator.train(training_data=training_data)

[lostella]

@SkanderHn there’s a few things that you can try:

• try setting pick_incomplete=False in DeepAREstimstor: this will affect how training windows will be sampled from the training data, avoiding sampling windows that have any zero-padding in the conditioning range.
• try setting scaling=False.
• try setting lags_seq=list(range(24)): this will enforce the RNN to get as input, at each time step, the 24 previous observations. I’m not sure what the default lags_seq is for weekly data otherwise.
• try any combination of the above :-)
• swap the DeepAREstimator for something else, like the SimpleFeedForwardEstimator, and see if that works better: if that’s the case, something may be wrong in DeepAREstimator, otherwise it could be that something is wrong in NegativeBinomial.

[antonkiselevtdg]

I achieved best results for intermittent sales forecasting with NPTS and Prophet.

[rt3722]

@antonkiselevtdg how were you able to use NPTS to train and predict ?

[antonkiselevtdg]

@antonkiselevtdg how were you able to use NPTS to train and predict ?

Just as any other available algorithm:

Example:
from gluonts.model.npts import NPTSEstimator

estimator = NPTSEstimator(freq="W", prediction_length=4,kernel_type = "exponential", use_seasonal_model = True)
predictor = estimator.train(training_data=training_data)

[lostella]

@SkanderHn are you giving the same training data as input to the predictor to get the forecasts? Could you include a minimal working example that reproduces the experiment?

[konradsemsch]

@lostella - when you write:

try setting pick_incomplete=False in DeepAREstimstor

how exactly should this be passed into the DeepAREstimator()? Do we need to recreate the DeepAREstimator transformation function with 'create_transformation' where we change some particular elements? How then can we pass those modifications into the actual DeepAREstimator?

From the Extended tutorial I can see that we transformation function can be applied on the data in GluonTs format, but can then the new dataset be passed into the estimator as is?

[lostella]

@lostella - when you write:

try setting pick_incomplete=False in DeepAREstimstor

how exactly should this be passed into the DeepAREstimator()?

I take that back, there's no such an option exposed

From the Extended tutorial I can see that we transformation function can be applied on the data in GluonTs format, but can then the new dataset be passed into the estimator as is?

I'm not sure I understand, could you be more specific?

[konradsemsch]

Sure, that's actually related to my other question here: link.

Let's say I would like to modify the default transformation function of DeepAREstimator and impact the way InstanceSplitter works, e.g.: change pick_incomplete or select a different sampling method. In the ExtendedTutorial it's mentioned how to create those new transformation functions and how to apply them to the data. However, it doesn't say how to pass that new transformation function/ transformed dataset into the estimator.

Let's say, if I create a new iterable object as a result of my modified transformation function, how could I train DeepAREstimator on it without using it's in-built default transformations?

Or otherwise, how to modify the default transformation function of DeepAREstimator and apply it to the training data during model training?

[lostella]

Let's say, if I create a new iterable object as a result of my modified transformation function, how could I train DeepAREstimator on it without using it's in-built default transformations?

Or otherwise, how to modify the default transformation function of DeepAREstimator and apply it to the training data during model training?

I'm afraid that there is no option to change the transformation without touching the code of the estimator class. Essentially you need to customize what this method returns: https://github.com/awslabs/gluon-ts/blob/master/src/gluonts/model/deepar/_estimator.py#L200

You can, for example, subclass the estimator and specialize the method with your own custom transformation.

Does this answer your question in the other thread?

[konradsemsch]

I'm not yet a python expert so I'm not 100% sure how to go about that, but in principle yes - that answers the question!

[fernandocamargoti]

@konradsemsch, did you manage to solve it?

[fernandocamargoti]

I'm also training for count series and the loss wasn't converging. I've seen this issue and found the PR https://github.com/awslabs/gluon-ts/pull/719 from @kashif. Since it wasn't merged yet, I forked the fixed distribution and trained again. Now my loss is converging.

[lostella]

@SkanderHn @fernandocamargoti #719 and #814 introduced fixes to the scaling issues with NegativeBinomial: could you try on the current master branch whether these help? (Especially with the example in the original post)

[konradsemsch]

I also checked that yesterday after I pulled the master branch but I haven't observed a particular improvement TBH... but perhaps I wasn't able to fully reproduce my last trials, so don't quote me on this ;)

I had a bit of a break from using gluon and now I'm coming back to it again, so I'll try to report again in case I find something.

[konradsemsch]

Ok, I can make a final confirmation today that the problem persists on my side after the update... Funny enough, this error wasn't coming before on my side. I'm using the M5 dataset here.

Initialisation of my estimator:

from gluonts.model.deepar import DeepAREstimator
from gluonts.distribution.neg_binomial import NegativeBinomialOutput
from gluonts.trainer import Trainer

estimator = DeepAREstimator(
    freq = metadata['freq'],
    prediction_length = prediction_horizon,
    context_length = prediction_horizon,
    num_layers = 2,
    num_cells = 40,
    dropout_rate = 0.1,
    use_feat_dynamic_real = True,
    use_feat_static_cat = True,
    cardinality = feat_static_cat_cardinalities,
    embedding_dimension = [min(50, (cat+1)//2) for cat in feat_static_cat_cardinalities], # [3050, 8, 4, 11, 4] - this is just for my reference
    distr_output = NegativeBinomialOutput(),
    # lags_seq = ...,
    # time_features = ...,
    trainer = Trainer(
        learning_rate = 1e-3,
        epochs = 1,
        batch_size = 32,
        num_batches_per_epoch = 50,
        hybridize = False # tried with False/ True
    )
)

predictor = estimator.train(train_ds)

The error that I'm seeing:

GluonTSUserError                          Traceback (most recent call last)
<ipython-input-15-57c4a0073b60> in <module>
     26 )
     27 
---> 28 predictor = estimator.train(train_ds)

~/.local/share/virtualenvs/Gluon-q9NlSOjv/lib/python3.7/site-packages/gluonts/model/estimator.py in train(self, training_data, validation_data, num_workers, num_prefetch, **kwargs)
    250     ) -> Predictor:
    251         return self.train_model(
--> 252             training_data, validation_data, num_workers, num_prefetch, **kwargs
    253         ).predictor

~/.local/share/virtualenvs/Gluon-q9NlSOjv/lib/python3.7/site-packages/gluonts/model/estimator.py in train_model(self, training_data, validation_data, num_workers, num_prefetch, **kwargs)
    229             input_names=get_hybrid_forward_input_names(trained_net),
    230             train_iter=training_data_loader,
--> 231             validation_iter=validation_data_loader,
    232         )
    233 

~/.local/share/virtualenvs/Gluon-q9NlSOjv/lib/python3.7/site-packages/gluonts/trainer/_base.py in __call__(self, net, input_names, train_iter, validation_iter)
    326                         if best_epoch_info.epoch_no == -1:
    327                             raise GluonTSUserError(
--> 328                                 "Got NaN in first epoch. Try reducing initial learning rate."
    329                             )
    330 

GluonTSUserError: Got NaN in first epoch. Try reducing initial learning rate.

On the other hand, I then tested the PoissonOutput as well as the default distribution and keep seeing the same message... do you know where I could commit an error and where to check first?

[konradsemsch]

Sorry, never mind - I forgot that I have included an additional covariate with missing values which seems to be causing the problem. After removing it things seem to be working fine! Would your recommendation be to impute such variables before training?

[kashif]

@konradsemsch yes so for missing covariates either impute it from past seasonal means etc. or you need to also predict it like the target...

[mbohlkeschneider]

DeepAR should run out-of-the box with missing values without the need to impute. You just have to encode missing values as NaN. The model should take it from there.

Edit: Sorry, missed covariates. The above only holds for target values. As @kashif already mentioned, covariates need to be imputed by the user.

[lostella]

After the recent changes, I trained DeepAREstimator on the data from the original post, with prediction length 52 (1 year), for 5 epochs and default settings otherwise (in particular, scaling is enabled). The predictions I get look like follows:

This could probably be improved, but the median predictions looks at least reasonable to me (the weird issues originally observed seem to be gone).

I'm closing this for now, please feel free to comment or reopen in case the issue still occurs.

[mariosantosprivate]

Sorry, never mind - I forgot that I have included an additional covariate with missing values which seems to be causing the problem. After removing it things seem to be working fine! Would your recommendation be to impute such variables before training?

Hi I'm having the same problem using a different set. When you say covariate do you mean one of the static category features?