Sktime: Data set simulators/generators

[mloning]

As discussed in #320, it would be great to have functions that generate/simulate data sets, they are useful for testing and explaining the difference between estimators.

• We currently have a very simple data set generator here: make_classification_problem
• TSML has some here in Java: https://github.com/uea-machine-learning/tsml/tree/master/src/main/java/statistics/simulators

[Kerybas]

I am interested in this issue.
I would like to propose the following approach:

1. Generate a bunch of randomised sine waves (10 here):
   
2. Pass them through a sklearn MLPRegressor randomly fitted, to obtain a number of "observations":
   
3. Optionally, discretize some of the signals in 2. to obtains classes (here, the red signal is discretized in 3 bins via KBinsDiscretizer):
   

Step 2. should ensure that the data set is complex while keeping a realistic relationship between the signals.

We could also add some noise, under-sampling, etc. to step 2 to make it even more realistic.

What do you think?
Is it worth investigating that approach?

[mloning]

@ABostrom @TonyBagnall any thoughts on this?

[ABostrom]

There's an arxiv paper did ages ago here: https://arxiv.org/abs/1703.09480

An old SimulationExperiment class from tsml: https://github.com/uea-machine-learning/tsml/blob/master/src/main/java/experiments/SimulationExperiments.java

I think our angle for simulation was, what type of problem are you looking to simulate within TSC.

Can you design a simulated problem that an interval classifier should be best on, same for DTW, dictionary, shapelet etc.

[fkiraly]

Hm, @TonyBagnall, @Kerybas, I thought it would be useful to get a quick way to generate synthetic test datasets for the different tasks, e.g., time series classification, regression, forecasting, that sktime supports?

Less about "methodological" side, but simply having it implemented for sktime test use?

[mloning]

Simulators for discriminating between classifiers would be useful for the part of the user guide #361 where we want to guide algorithm choice and explain the inner workings of the different algorithms.

[TonyBagnall]

so I wrote the data simulator to reinforce the design mechanism of hive-cote, i.e. that if you have no prior knowledge as to the correct feature space, a meta ensemble is the best approach. A simulator that just creates a task problem is much simpler, simply generate white noise. However, I would like to use more simulation experimentation, not least to satisfy the statisticians. I think we could definitely add this to the new road map we are developing, and make it a project, once we have prototyped the code structure

[mloning]

Gluon-ts also have a lot of simulation functionality for forecasting/anomaly detection: https://gluon-ts.mxnet.io/examples/synthetic_data_generation_tutorial/tutorial.html

[sjmiller8182]

I have existing code for simulating univariate ARMA, ARIMA time series based on a given polynomial. Would that be of interest here?

[mloning]

@sjmiller8182 yes certainly, the discussion so far has focussed on time series classification, but having data generators for forecasting would be very helpful too! 👍

I'd appreciate a draft PR with a new file for generators in sktime/datasets/. Also feel free to open a separate issue to discuss this in more detail if you have more questions!

FYI statsmodels has an ARMAProcess generator too.