Gluon-ts: issue with repository datasets

[kashif]

Description

I noticed some issues with the repository datasets, e.g. the taxi_30min dataset's test set has 57 times more entities than the train! Is that intended? Also the this dataset's test time series is only 149 days in 2016, whereas the train is 1488 days starting in 2015. Can you kindly clarify this thanks!!

Other datasets's test also seem to have some whole number multiple more timeseries entities than the train. I can paste the numbers here you want, just let me know.

To Reproduce

(Please provide minimal example of code snippet that reproduces the error. For existing examples, please provide link.)

Error Message

(Paste the complete error message, including stack trace.)

Environment

• Operating system:
• Python version:
• GluonTS version:

(Add as much information about your environment as possible, e.g. dependencies versions.)

[mbohlkeschneider]

Hi @kashif,

Thank your for this question, I'll try to elaborate here:

1. The train and test data are from January 2015 and January 2016, respectively, so this is intended. We tried to be close to the source that we cited in our paper, which uses this setup.

2. The granularity of the data is 30 minutes, so January has 31 (days) * 24 (hours) * 2 (twice per our) = 1488 time points in train, which is intended.

3. The test set has 57 "more" time series because of the rolling evaluation setup. Basically, we choose one start date to start the evaluation and then "roll" the evaluation window by prediction_length to evaluate the same model over all of January. Technically, January would have 1488/24 = 62 windows, but we need some time steps as context_length + the used lags. Thus, we only use 57 windows to accommodate this extra data we need for the first "window". You will note that the time series 1215 is the first series, but extended by 24 time steps for the rolling evaluation. This also applies to the other datasets, but with less rolling windows.

4. The only thing I cannot wrap my head around is why the first window of the first time series is 149 time steps. The longest lag we use is 48, so we would only need an initial length of 48 + 24 (context length) + 24 (prediction length). Maybe this was longer in early iteration where we used different lags? I'll reach out to the co-author that processed that dataset to clarify that.

So most of the numbers make sense to me and are intended.

Maybe two pointers why this is implemented this way:

1. Our evaluator cuts the last prediction_length time points of the test time series for evaluation. Thus, the last prediction_length time points are always the ones that are evaluated.

2. We don't have a module that implements rolling backtests by doing the splitting and evaluation for you, so we manually processed the test datasets to mimic this rolling backtest setup.

P.S.: Details in dataset processing and experimental setup are in section 5 of the main text and section F in the appendix in our paper.

Hope that helps.

[kashif]

thanks @mbohlkeschneider for the answers. looking forward to point no. 4.

[kashif]

@mbohlkeschneider slightly unrelated question, but can you tell me how then using the MultivariateGrouper one evaluates and runs the backtesting on test data? Thank you in advance! If you can point me to some example code that would be awesome!!

[mbohlkeschneider]

Hi @kashif,

I reached out to our co-author and as I suspected we have been playing with different aggregations/lags in the taxi dataset in the initial exploration phase. The 149 time steps in the test data is an artifact from that covering the required history. We should have been more precise in the paper to exactly mention the time slice that we use in the train/test split.

One small issue that results from that is there is an overlap of five time steps in the last and second to last validation window (i.e. five time steps are evaluated twice). However, I don't think this influences the findings and should have close to zero influence on the experiments.

One example on how to run multivariate models can be found in our release branch of the multivariate model. This example is a bit convoluted, though. Hope it helps anyway!

[kashif]

thanks @mbohlkeschneider