Sktime: [BUG] AutoARIMA not working with Temporal Cross Validation

[ngupta23]

Describe the bug
Would like the statistical models like ARIMA, ETS, etc to work with the Temporal Cross Validation Flow. I am not able to reproduce the same results as the standalone ARIMA when using the Temporal Cross Validation Flow.

To Reproduce

Setup

y = load_airline()
y_train, y_test = temporal_train_test_split(y, test_size=36)

fh = ForecastingHorizon(np.arange(len(y_test)) + 1, is_relative=True)

Regular AutoARIMA (Baseline for test)

forecaster = AutoARIMA(sp=12, suppress_warnings=True)
forecaster.fit(y_train)
y_pred = forecaster.predict(fh)
plot_series(y_train, y_test, y_pred, labels=["y_train", "y_test", "y_pred"]);
smape_loss(y_test, y_pred)

0.04117062370076287

Using Temporal Cross Validation

Version 1: 'sp' in forecaster_param_grid (does not work)

forecaster_param_grid = {'sp': [12]}
forecaster = AutoARIMA(suppress_warnings=True)

cv = SlidingWindowSplitter(initial_window=int(len(y_train) * 0.90), start_with_window=True)
gscv = ForecastingGridSearchCV(forecaster, cv=cv, param_grid=forecaster_param_grid, verbose=True)

gscv.fit(y_train)
y_pred = gscv.predict(fh)
plot_series(y_train, y_test, y_pred, labels=["y_train", "y_test", "y_pred"]);
smape_loss(y_test, y_pred)

0.11346208431398466

Version 2: 'sp' in forecaster (works but defeats the purpose of Grid Search)

forecaster_param_grid = {}
forecaster = AutoARIMA(sp=12, suppress_warnings=True)

cv = SlidingWindowSplitter(initial_window=int(len(y_train) * 0.90), start_with_window=True)
gscv = ForecastingGridSearchCV(forecaster, cv=cv, param_grid=forecaster_param_grid, verbose=True)

gscv.fit(y_train)
y_pred = gscv.predict(fh)
plot_series(y_train, y_test, y_pred, labels=["y_train", "y_test", "y_pred"]);
smape_loss(y_test, y_pred)

0.04117062370076287 (matches standalone AutoARIMA without GridSearch above)

Expected behavior
It does not look like the best_estimator is taking the seasonality value of 12 when 'sp' is just passed through the forecaster_param_grid. It only works if it is set natively in the forecaster initialization.

Additional context
Basically, I would like to create a unified flow around sktime to build and compare multiple models (ARIMA, ETS, Random Forest, SVM, etc), including hyper parameter parameter for the statistical models. I see from the examples folder how this can be done for native scikit models but wanted to recreate the same for the statistical models

Versions

System:
python: 3.6.12 |Anaconda, Inc.| (default, Sep 9 2020, 00:29:25) [MSC v.1916 64 bit (AMD64)]
executable: C:\Usersxxxx\AppData\Local\Continuum\anaconda3\envs\sktime\python.exe
machine: Windows-10-10.0.18362-SP0

Python dependencies:
pip: 20.3
setuptools: 49.6.0
sklearn: 0.23.2
numpy: 1.19.2
scipy: 1.5.2
Cython: 0.29.17
pandas: 1.1.3
matplotlib: 3.3.2
joblib: 0.17.0
numba: None
pmdarima: 1.7.1
tsfresh: None

[mloning]

There may be a bug here, I'm not sure though what it is. Why do you think there is a bug here?

Note that AutoARIMA and GridSearchCV use different procedures to pick the "best" set of parameters. You also use different training sets (the full y_train in AutoARIMA vs only the initial 90% of y_train in GridSearchCV).

If you're interested in comparative benchmarking for forecasting, take a look at our code and paper here: https://github.com/mloning/sktime-m4

If you end up running more experiments, we're happy to collaborate on this!

[ngupta23]

Hi @mloning, Thanks for the reference to the repo! I will go through it and let you know if I have any further questions.

[mloning]

Great - I'll close this issue for now but feel free to reopen it again if anything is in fact broken!

[ngupta23]

I think I figured out the issue. The issue is that the AutoARIMA module is not honoring the set_params internally. It is getting updated in the AutoARIMA class itself since it is inheriting from the base sklearn class, but the updated parameters are not getting passed to the internal _forecaster.

Expected Behavior

regressor = KNeighborsRegressor()
print(f"Original n_neighbors: {regressor.get_params()['n_neighbors']}")

regressor.set_params(n_neighbors=1)
print(f"\nAfter manually setting n_neighbors: {regressor.get_params()['n_neighbors']}")

Original n_neighbors: 5

After manually setting n_neighbors: 1

As we can see the parameters get updated

Unexpected Behavior

forecaster = AutoARIMA(suppress_warnings=True)
print(f"Original sp: {forecaster.get_params()['sp']}")
print(f"Original _Forecaster: {forecaster._forecaster}")
print(f"Original _Forecaster sp: {forecaster._forecaster.m}")

forecaster.set_params(sp=3)
print(f"\nAfter manually setting sp: {forecaster.get_params()['sp']}")
print(f"After manually setting sp _Forecaster: {forecaster._forecaster}")
print(f"After manually setting sp _Forecaster sp: {forecaster._forecaster.m}")

Original sp: 1
Original _Forecaster: AutoARIMA(error_action='warn', suppress_warnings=True, with_intercept=True)
Original _Forecaster sp: 1

After manually setting sp: 3
After manually setting sp _Forecaster: AutoARIMA(error_action='warn', suppress_warnings=True, with_intercept=True)
After manually setting sp _Forecaster sp: 1

In this case, the parameters get adjusted in the AutoARIMA class, but are not passed to the underlying _AutoARIMA object (remains set to the default value of 1).

Issue that it causes

So when we use the ForecastingGridSearchCV with AutoARIMA, the parameter sp does not get updated at all during the set_param step outlined below and remains set to the default value of 1.

https://github.com/alan-turing-institute/sktime/blob/d3ac2b45d89946363899370c3b43cf52a103e501/sktime/forecasting/model_selection/_tune.py#L119

This why all the results come out identical from the hyperparameters search as can be seen from the metrics below (mean sMAPE = 0.40163609 for both sp=3 and sp=12.

forecaster_param_grid = {'sp': [3, 12]}
forecaster = AutoARIMA(suppress_warnings=True)

cv = SlidingWindowSplitter(
    initial_window=int(len(y_train) * 0.5),
    start_with_window=True,
    window_length=36,
    # fh=np.arange(12) + 1  # Not Supported Yet (Prediction can only be the next time point for now)
)

gscv = ForecastingGridSearchCV(forecaster, cv=cv, param_grid=forecaster_param_grid, verbose=True)
gscv.fit(y_train)
gscv.cv_results_

```
{'mean_fit_time': array([1.60882616, 1.63023949]),
'mean_score_time': array([0.18559933, 0.14276004]),
'param_sp': masked_array(data=[3, 12],
mask=[False, False],
fill_value='?',
dtype=object),
'params': [{'sp': 3}, {'sp': 12}],
'mean_test_sMAPE': array([0.40163609, 0.40163609]),
'rank_test_sMAPE': array([1, 1])}

### Potential Solution

The solution has been derived using the recommendations found on [this page](https://scikit-learn.org/stable/developers/develop.html). In particular, a custom set_param() method could be written in the AutoARIMA class to also override the _Forecaster param values

```python
def set_params(self, **parameters):
    for parameter, value in parameters.items():
        setattr(self, parameter, value)
        if parameter == 'sp':
            parameter='m'
        setattr(self._forecaster, parameter, value)  ## Added to propagate the parameter to the underlying _Forecaster
    return self

I have verified this locally and it works. If you are ok with it, I can submit a PR.

forecaster_param_grid = {'sp': [3, 6, 12, 24]}
forecaster = AutoARIMA(suppress_warnings=True)

cv = SlidingWindowSplitter(
    initial_window=int(len(y_train) * 0.5),
    start_with_window=True,
    window_length=36,
    # fh=np.arange(12) + 1  # Not Supported Yet (Prediction can only be the next time point for now)
)
gscv = ForecastingGridSearchCV(forecaster, cv=cv, param_grid=forecaster_param_grid, verbose=True)
gscv.fit(y_train)
gscv.cv_results_

{'mean_fit_time': array([0.67233038, 0.76242065, 4.10686517, 0.00775552]),
 'mean_score_time': array([0.11560488, 0.15221286, 0.1613667 , 0.        ]),
 'param_sp': masked_array(data=[3, 6, 12, 24],
              mask=[False, False, False, False],
        fill_value='?',
             dtype=object),
 'params': [{'sp': 3}, {'sp': 6}, {'sp': 12}, {'sp': 24}],
 'mean_test_sMAPE': array([0.37408572, 0.37408572, 0.29277678,        nan]),
 'rank_test_sMAPE': array([2, 2, 1, 4])}

Now the results vary for each value of the hyperparameter as expected.

[mloning]

I see - thanks @ngupta23 for investigating this!

I think there's a simpler solution. The problem is that we instantiate self._forecaster in __init__, instead we should instantiate it in fit. That way, updated params would be propagated properly. What do you think?

https://github.com/alan-turing-institute/sktime/blob/d3ac2b45d89946363899370c3b43cf52a103e501/sktime/forecasting/arima.py#L306

[ngupta23]

Hi @mloning, Thanks you for suggesting the alternative. You will need to change the argument assignments in _AutoARIMA from just to self. and then it will pick the updated param value after assignment is done in GC. It is definitely cleaner since we don't have to rename the parameters such as sp.

Would you like me to submit a PR for this?

[mloning]

Yes that would be awesome!