Gluon-ts: Changes when switching between DeepAR, DeepFactor, DeepStateSpace?

[andmib]

Hello all,

I've been utilizing DeepAR for forecasting individual sales of products. I've been testing out DeepFactor and DeepState estimators as well, and despite the substantial improvements over DeepAR seen in the papers, both DeepFactor and DeepState seem to do a lot worse overall when comparing with the same number of batches/epochs and default layers/cells.

Are there any specific modifications to the data or network (i.e. "works better with more training data", or "tuning layers is necessary") I should be making when trying out DeepFactor and DeepState?

[rshyamsundar]

First of all note that this (deepstate) is a reimplementation of the paper code using the gluon interface of mxnet. The code currently does not fully expose all the hyper-params used in the paper and the defaults for other hyper-params might not be optimal for all datasets.

Having said that there might be several reasons for bad performance:

1. Is your target variable (sales) integer-valued? Deep state has strong assumption that the output follows Gaussian distribution and is not a good model for count data.

2. Have you enabled features, esp. categorical features for DeepStateEstimator? By default, none of the estimators use features even if they are present in the dataset. For deepar which uses lags this might not be that problematic, but for deepstate this can affect results severely.

3. As mentioned, some hyper-params are not exposed in the open source version, e.g. prior covariance, upper bound on noise and innovation variance. We are working on updating this. Tuning this might help improve performance.

[andmib]

1. Is your target variable (sales) integer-valued? Deep state has strong assumption that the output follows Gaussian distribution and is not a good model for count data.

It is indeed count data, and does not follow a Gaussian distribution. That could explain it!

2. Have you enabled features, esp. categorical features for `DeepStateEstimator`? By default, none of the estimators use features even if they are present in the dataset. For deepar which uses lags this might not be that problematic, but for deepstate this can affect results severely.

Yes, I do have static categorical features enabled for both DeepAR and DeepState - they are somewhat critical to the forecast because they describe the product grouping.

3. As mentioned, some hyper-params are not exposed in the open source version, e.g. [prior covariance](https://github.com/awslabs/gluon-ts/blob/695a226ec9d323a49f283c6f9d1b8ccc32fdfa86/src/gluonts/model/deepstate/_network.py#L67), [upper bound on noise and innovation variance](https://github.com/awslabs/gluon-ts/blob/695a226ec9d323a49f283c6f9d1b8ccc32fdfa86/src/gluonts/distribution/lds.py#L476-L477). We are working on updating this. Tuning this might help improve performance.

Looking forward to it!

Do any of these assumptions also apply for DeepFactor?