Gluon-ts: GluonTS M4 benchmark code produces M4 results inferior to M4 competition results

[gjmulder]

I'm smoke testing your ./gluon-ts/examples/benchmark_m4.py on some GPUs, and looking at the point forecasts for the different frequencies. Here's some sample results from the Monthly M4 data, with comparisons to both the statistical benchmarks and the two top M4 Competition entrants:

  |   | sMAPE | MASE
-- | -- | -- | --
benchmark | seasonal_naive | 15.256 | 1.205
  | naive | 15.988 | 1.260
  | naive2 | 14.427 | 1.063
  | ses | 13.618 | 1.020
  | holt | 14.828 | 1.010
  | theta_classic | 13.473 | 0.972
  | holt_damped | 13.003 | 0.970
  | combined | 13.434 | 0.966
  |   |   |  
GluonTS | DeepAREstimator | 14.067 | 1.145
  | MQCNNEstimator | 14.444 | 1.173
  |   |   |  
Smyl | Slawek | 12.126 | 0.884
Montero-Manso | Pablo | 12.639 | 0.893

I seem to remember @rshyamsundar mentioning in the ISF workshop in Thessaloniki that the GluonTS results were competitive with the M4 entrants but, assuming I'm performing an apples-to-apples comparison, benchmark_m4.py doesn't seem to be @rshyamsundar was referring to. If so, would you have any tuning / config / methodology suggestions for producing better results with GluonTS?

[tm1611]

Hi, I was also wondering how competitive the DeepAR algorithm is vs. the M4 methods as well as how more recent methods such as DeepState and DeepFactor perform in comparison. So here a few points:

I played around with the algorithm and data and found that for example increasing the number of epochs or num_batches_per_epoch parameter increases the performance of DeepAR considerably. M4_monthly consists of 48,000 series so it seems the algorithm needs to "see" a little bit more data. For your example of the m4_monthly data (num_batches_per_epoch=200) that leads to MASE of 0.9972 and sMAPE of 13.556 which is still far away from Smyl and Montero-Manso.

In [1] you can find an aggregated accuracy of DeepAR on the entire M4 data (MASE=1.500, sMAPE=0.1192) vs Smyl (MASE=1.54, sMAPE=0.1137) and it states that "Its accuracy is close to the winning solution in the M4 competition" using default hyper-parameters so this may be the source of the competitiveness claim. However, there is also a built-in DeepAR algorithm in AWS SageMaker that, according to the gluonts.model.deepar documentation, "is unrelated to the implementation behind [SageMaker]". From the paper it is not clear to me whether the gluonts.model.deepar implementation or the SageMaker implementation was used.

For comparison, according to my own calculations the automatic exponential smoothing function (ETS) of the R forecast package yields a sMAPE of 13.5356 and MASE of 0.945. The automatic ARIMA function yields similar results and simply averaging both methods works even better (sMAPE=12.9769, MASE=0.9092) for m4_monthly data. Results are calculated on my own and not validated (just saying as I am a little skeptical on the performance of this simple ETSARIMA approach). I know that DeepAR is not conceptualized for such a heterogeneous dataset as the M4 data which seems to be a possible explanation.

My specifications for DeepAR on m4_monthly: {mx.random.seed(42), num_batches_per_epoch=200} and using the defaults for other hyperparameters. Moreover, neither DeepFactor nor DeepState seem to be close to being competitive on the M4 data using their defaults. It would be nice to know why this is the case and possible adjustments on their specifications.

Lastly, stability of results is definitely an issue as running the same model twice without seed yields highly variable results. Hope this was helpful.

[1] T. Januschowski, J. Gasthaus, Y. Wang et al., Criteria for classifying forecasting methods. International Journal of Forecasting (2019)

[gjmulder]

I've recently performed some GPU-based hyper-parameter optimisation on a modified M4 Daily data set:

• Added a static categorical for the domain (Demographic, Finance, Industry, Macro, Micro, Other)
• Added a static categorical indexing the individual time series (as per #410)

There is an open issue #336 for running DeepState models on GPU, so I've focused my search on DeepAR models. My latest DeepAR hyper-parameters are:

• "batch_size" : 32,
• "dropout_rate" : 0.0880,
• "learning_rate" : 0.00625,
• "learning_rate_decay_factor" : 0.406,
• "max_epochs" : 60000,
• "minimum_learning_rate" : 8.32e-06,
• "num_batches_per_epoch" : 60,
• "num_cells" : 100,
• "num_layers" : 4,
• "weight_decay" : 1.76e-8,
• "patience" : 50,

And the results found at epoch 59558 were:

• 'MASE': 3.218
• 'sMAPE': 3.035

While better than the M4 Competition overall winning entries, this is extremely close to the result from the statistical methods, so a lot of GPU cycles spent for no appreciable gain :-)

I'll look at the M4 Hourly data set next, which was what @rshyamsundar et al reported their results against for the Deep State paper.

[tm1611]

Thank you very much @gjmulder , I am looking forward to additional results from your side and would like to help in contributing some more results on the M4 data if help is wanted.

So far, the best results that I got was an MASE of 3.3355 (including seeds=44 in m4_benchmark.py, epochs=200, num_batches_per_epoch=200, defaults for everything else) on the m4_daily.

Unfortunately, I could not replicate your results because I run into two problems and would be highly interested in your knowledge regarding

1) How you included two static categorical variables in the algorithms and
2) how you implemented HPO with gluonTS?

It would be nice if you'd be willing to share some of your knowledge, provide example code / an example script or something like that.

Also, the m4_daily are dominated by data of the "finance" domain (37% of the daily data are from this area) for which naive methods, such as the Naive2, work suprisingly well (multiplicative decomposition seems to work well on data with high ACF1). Hence, it may be interesting to compare the m4_daily by subsets (for example finance vs. non-finance) against benchmarks and other m4 methods.

Thanks a lot.

[gjmulder]

The m4_daily look to be hard to forecast using ML methods. In contrast, the m4_hourly seems to be extremely challenging for the structural/statistical methods, perhaps due to multiple seasonality (24 hour and 7 day).

Here's the R script that I'm using to (re)generate the JSON M4 data. It assumes the metadata and directory structure in ~/.mxnet already exists and the M4-info.csv file (https://github.com/M4Competition/M4-methods/blob/master/Dataset/M4-info.csv) is present in the current dir. There's a bit of date logic I needed to implement to deduce valid start dates as M4-ifo.csv is full of Y2K bugs (!!):

split_M4_training.R.txt

Note that the split code runs in two modes. "M4 competition" mode where it splits the M4 competition data. And "final" mode where it uses the complete post-competition data set. It is incorrect to compare accuracies with the M4 competition winners using the full data set, especially if you're building 100's of models, as I am.

The idea is that the categorical index of the individual TS is useful for local fitting, while the domain category of the TS might help global/meta-learning as per Smyl's winning approach. Getting start_dates correct ensures that the time series are aligned optimally for the lags GluonTS automatically generates based on the frequency of the time series.

My HPO code is a bit in flux. I'll have a useful self-contained and stable example soonish. I'm using HyperOpt (https://github.com/hyperopt/hyperopt) which in turn uses MongoDB to orchestrate the HPO. I kick off multiple jobs across a farm of GPUs and HyperOpt pulls model candidate designs from MongoDB and writes back their results. It is great - I can sit there running MongoDB queries and get real-time results and trends on my HPO search.

[lixixibj]

I'm smoke testing your ./gluon-ts/examples/benchmark_m4.py on some GPUs, and looking at the point forecasts for the different frequencies. Here's some sample results from the Monthly M4 data, with comparisons to both the statistical benchmarks and the two top M4 Competition entrants:

    sMAPE MASE
benchmark seasonal_naive 15.256 1.205
  naive 15.988 1.260
  naive2 14.427 1.063
  ses 13.618 1.020
  holt 14.828 1.010
  theta_classic 13.473 0.972
  holt_damped 13.003 0.970
  combined 13.434 0.966
       
GluonTS DeepAREstimator 14.067 1.145
  MQCNNEstimator 14.444 1.173
       
Smyl Slawek 12.126 0.884
Montero-Manso Pablo 12.639 0.893
I seem to remember @rshyamsundar mentioning in the ISF workshop in Thessaloniki that the GluonTS results were competitive with the M4 entrants but, assuming I'm performing an apples-to-apples comparison, benchmark_m4.py doesn't seem to be @rshyamsundar was referring to. If so, would you have any tuning / config / methodology suggestions for producing better results with GluonTS?

Hi, did you try methods in gluon-ts on other datasets of M4 competition? Could you please post the forecasting accuracy of other datasets online? I want to compare my new method with gluon-ts on M4 datasets. Thanks a lot.

[tm1611]

@gjmulder

Thanks again for your contribution and I am looking forward to more from your side. A few related notes (if not otherwise mentioned, I am refering to DeepAR):

1. Regarding your R script:

I have slightly modified your script so it only uses the last 300 observations for yearly data. This is similarly to how yearly data are handled in gluonts to make numpy happy https://github.com/awslabs/gluon-ts/blob/481fc5fdd09abad012cf6398e06bc6be8b1de197/src/gluonts/dataset/repository/_m4.py#L60

Easy fix in R:

if (length(target) >= 300){
    target <- xts::last(target, 300)
}

Also, I incorporated the possibility to opt for the same gluonts mock start date so true_dates vs. mock date can be compared.

#
  if (true_dates == FALSE){
    start <- "1750-01-01 00:00:00"
  } else {
    start <- paste0(as.character(as.Date(time(train))[1]), " 00:00:00") 
  }

2. true_dates vs. mock_date ("1750-01-01 00:00:00").

Referring to:

Getting start_dates correct ensures that the time series are aligned optimally for the lags GluonTS automatically generates based on the frequency of the time series.

• Experiment 1: m4_yearly_true_dates vs. m4_yearly (mock dates)
• Experiment 2: m4_quarterly_true_dates vs. m4_quarterly (mock dates)
• Experiment 3: m4_monthly_true_dates vs. m4_monthly (mock dates)

Simple t-test on the means of MASE/sMAPE indicates that there is no difference between accuracy of using true dates vs mock dates, c.p., and based on twenty trials per experiment (10 vs. 10) given default hyperparameters. This is consistent across all three tested frequencies.

[gjmulder]

@tm1611 I finally had some opportunity over Christmas to get back to this. Interestingly, I have found that providing the correct start dates for the daily data does provide a noticeable improvement, specifically in MASE which is perhaps more sensitive to seasonality error:

Model|Data Set|Training MASE|Training sMAPE|Testing MASE|Testing sMAPE
-|-|-|-|-|-
DeepAREstimator|m4 daily, mock dates|4.373|37.70 |4.546|41.34
DeepAREstimator|m4 daily, true dates|4.291|36.57|4.304|39.19

(Note that these results are preliminary and represent different model architectures found by HPO, so not as apples-to-apples as your tests).

I suspect that for yearly, quarterly and monthly there's not much in the way of seasonal knowledge transfer between the time series. In contrast, the daily data has multiple seasonalities (weekly, monthly and yearly), so the correct date alignment for daily data may allow it to learn things like weekend and holiday effects, etc.